text:
Missing documentation regarding the anatomical definitions of the 468 face landmarks

my student and I are working on a robust registration between face-mesh with other head/brain-landmarks used for neuroimaging, specifically, the 10-20 system for EEG.
the comments in a previous issue (#1615) seem to give a good picture how these face landmarks are numbered, but I still don't see a rigorous definition on where these key points are supposed to be from a face-feature perspective, for example, where is landmark 10 is supposed to be located? middle of the forehead? is there a standard or document where I can find face landmark's "actual" geometric anatomical definitions?
thanks

Hi @fangq,
Is it implemented through Python, Javascript or other solutions available in MediaPipe? Could you please provide the complete details to understand the issue better from our end. Thank you!

Currently, our initial implementation is based on python, but the question is more generic and independent of implementation.
on a related note - among the 468 face mesh points, are there "dependent" points that are computed/interpolated based on other independently estimated points? if so, what are those and how they are interpolated?

I'm not sure if this helps at all, but there is a .obj file located here in the MediaPipe repository. You can open this file in external software to visualize and analyze the face mesh model. For example, using MeshLab, you can get the index of a vertex in a mesh as shown below:

To do so:

Open MeshLab, which will create a new, empty project.
Drag & drop canoncal_face_model.obj onto the window, which will import the mesh into this new project.
Click the "Get Info" button (yellow circle with an "i" in the icon bar).
Press "t" to switch from face-selection mode (default) to vertex-selection mode.
Click any vertex to get its index.

I am very much a beginner when it comes to MeshLab, but if you have any questions, I would be happy to try and answer them. I recommend playing around with the vertex display size as shown in my screenshot above, on the right, in the sidebar.

@rajkundu thanks for your reply, but I don't think it answers the question I was asking - this still only gives the numbering of points, but not what face feature they are supposed to be pointing to.
just an example - what location on the face that point 10 is supposed to be?

Sorry; I'm not entirely sure what you mean. To me, it seems like that question is answerable by looking at the canonical face model UV map, then seeing where Point 10 is. Or, if you're interested in Point 10's 3D definition, I would look at the .obj file.
Perhaps you're looking for an anatomical definition instead?

Perhaps you're looking for an anatomical definition instead?

yes, that is correct. I previously referred it to as "geometric definition", but I think "anatomical definition" is more appropriate.

Ah, I see. To be honest, I'm not quite sure how these types of things tend to work. I think the question now is whether Google's engineers attached anatomical definitions to any of the the MP face model's points. I mean, to some extent, strict anatomical correspondences may not even be necessary for the model to function as intended.
My guess is as good as yours, but I would imagine the MP face model does not have many, if any, anatomical correspondences. Some of the only easy/robust ones that I can think of are: the corners of the eyes, the tip of the nose, and maybe even the nasion (vertex 168?). In general, I imagine that finding these correspondences is our job; I don't think there is an official document on them, though I could certainly be wrong.

yes, the only anatomically-clear landmark relevant to my project is nasion (pt168), other than that, I have been computing other 10-20 anchor points empirically by extrapolating 10, 234, 227, 457 and 447, which worked reasonably well, but I was hoping if all landmarks were anatomically defined, then I would get more assets to improve the mapping.

It always puzzle me to find out the the landmark id. So I injected .obj file with material section. And use a simple material file to bind the uv visualization map. It makes the obj could be opened in Mesh viewer with vertex id...

or you can import canonical face model to blender, then find interpolation weights to define your customized landmarks...

@kuaashish, thanks for following up. We are still looking for official documentation on these landmarks. My student had spent the past two  semesters trying to fit these facial landmarks to anatomically defined head landmarks using a large library, but if knowing more on their definitions will still help for refine this registration.

@fangq,
Thanks for the confirmation, For now, We do not have any anatomical naming scheme for each of the 468 landmark. However, We can mark this as feature request and share internally based on discussion, We will prioritise the request.

i'm searching for definitions too, and i just found this keypoints the orders of landmark indexes are mirrored e.g. in rightEyeUpper0 and leftEyeUpper0. But still lack of that on other facial parts like check or forehead.

authors: andrechen fangq kuaashish rajkundu rezinghost

repo: google/mediapipe


text:
Tulsi project does not generate; cannot find numpy

Please make sure that this is a build/installation issue and also refer to the troubleshooting documentation before raising any issues.
System information (Please provide as much relevant information as possible)

OS Platform and Distribution: Mac M1 chip, MacOS 12.6.1, attempting to use Tulsi to generate iOS
MediaPipe version: top of tree
Bazel version: using bazelisk. Bazel is currently at 5.3.2
XCode and Tulsi versions (if iOS): Xcode 13.4.1, Tulsi from top of tree
Python is at Python 3.10.6. I cannot use Python 3.7 on Mac M1 computers. I can switch to 3.8 if needed but I don't think that's the issue here.

Describe the problem:
Follow directions to build for iOS. On generating xcode project in Tulsi, get this:
ERROR: An error occurred during the fetch of repository 'local_execution_config_python':Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
ERROR: /Users/diglesia/mediapipe/WORKSPACE:488:14: fetching local_python_configure rule //external:local_execution_config_python: Traceback (most recent call last):Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
ERROR: Analysis of target '//mediapipe/objc:mediapipe_framework_ios' failed; build aborted: Problem getting numpy include path.Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
General failure: Bazel aspects could not be built.

Python is at
/opt/homebrew/bin/python
and it finds numpy correctly when used.

Some more logs:
[3:15:43 PM](3): Running /opt/homebrew/bin/bazel info '--override_repository=tulsi=/Users/diglesia/Library/Application Support/Tulsi/0.20221005.88/Bazel'
[3:22:35 PM](3): Running /opt/homebrew/bin/bazel --max_idle_secs=60 query '--override_repository=tulsi=/Users/diglesia/Library/Application Support/Tulsi/0.20221005.88/Bazel' --announce_rc --noimplicit_deps --order_output=no --noshow_loading_progress --noshow_progress 'kind(rule, :all)+kind(rule, mediapipe:all)+kind(rule, mediapipe/examples/ios:all)+kind(rule, mediapipe/examples/ios/facedetectioncpu:all)+kind(rule, mediapipe/examples/ios/facedetectiongpu:all)+kind(rule, mediapipe/examples/ios/faceeffect:all)+kind(rule, mediapipe/examples/ios/facemeshgpu:all)+kind(rule, mediapipe/examples/ios/handdetectiongpu:all)+kind(rule, mediapipe/examples/ios/handtrackinggpu:all)+kind(rule, mediapipe/examples/ios/holistictrackinggpu:all)+kind(rule, mediapipe/examples/ios/iristrackinggpu:all)+kind(rule, mediapipe/examples/ios/objectdetectioncpu:all)+kind(rule, mediapipe/examples/ios/objectdetectiongpu:all)+kind(rule, mediapipe/examples/ios/objectdetectiontrackinggpu:all)+kind(rule, mediapipe/examples/ios/posetrackinggpu:all)+kind(rule, mediapipe/examples/ios/selfiesegmentationgpu:all)+kind(rule, mediapipe/objc:all)' --output xml
[3:22:58 PM](3): Running /opt/homebrew/bin/bazel info '--override_repository=tulsi=/Users/diglesia/Library/Application Support/Tulsi/0.20221005.88/Bazel'
[3:22:58 PM](3): Build Events JSON file at "/var/folders/fm/yng8gs1s2kb1kwv6gvsv4nc4007656/T/build_events_65322.json"
[3:22:58 PM](3): Running /opt/homebrew/bin/bazel build --announce_rc --show_result=0 --noshow_loading_progress --noshow_progress --symlink_prefix=/ --apple_platform_type=ios --cpu=ios_x86_64 --watchos_cpus=i386 --announce_rc '--override_repository=tulsi=/Users/diglesia/Library/Application Support/Tulsi/0.20221005.88/Bazel' --compilation_mode=dbg --define=apple.add_debugger_entitlement=1 --define=apple.propagate_embedded_extra_outputs=1 --define=apple.experimental.tree_artifact_outputs=1 --features=debug_prefix_map_pwd_is_dot --tool_tag=tulsi:generator --build_event_json_file=/var/folders/fm/yng8gs1s2kb1kwv6gvsv4nc4007656/T/build_events_65322.json --noexperimental_build_event_json_file_path_conversion --noexpand_test_suites --features=-parse_headers --noexperimental_run_validations --aspects @tulsi//:tulsi/tulsi_aspects.bzl%tulsi_sources_aspect --output_groups=tulsi_info,-dsyms //mediapipe/examples/ios/facedetectioncpu:FaceDetectionCpuApp //mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp //mediapipe/examples/ios/faceeffect:FaceEffectApp //mediapipe/examples/ios/facemeshgpu:FaceMeshGpuApp //mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp //mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp //mediapipe/examples/ios/helloworld:HelloWorldApp //mediapipe/examples/ios/holistictrackinggpu:HolisticTrackingGpuApp //mediapipe/examples/ios/iristrackinggpu:IrisTrackingGpuApp //mediapipe/examples/ios/objectdetectioncpu:ObjectDetectionCpuApp //mediapipe/examples/ios/objectdetectiongpu:ObjectDetectionGpuApp //mediapipe/examples/ios/objectdetectiontrackinggpu:ObjectDetectionTrackingGpuApp //mediapipe/examples/ios/posetrackinggpu:PoseTrackingGpuApp //mediapipe/examples/ios/selfiesegmentationgpu:SelfieSegmentationGpuApp //mediapipe/objc:mediapipe_framework_ios
[3:22:59 PM](0): Failed to retrieve information about the Bazel workspace. This usually means that the contents of a BUILD file are incorrect or flags are missing. Read the message log carefully for details. [Details]: ERROR: An error occurred during the fetch of repository 'local_execution_config_python':Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
ERROR: /Users/diglesia/mediapipe/WORKSPACE:488:14: fetching local_python_configure rule //external:local_execution_config_python: Traceback (most recent call last):Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
ERROR: Analysis of target '//mediapipe/objc:mediapipe_framework_ios' failed; build aborted: Problem getting numpy include path.Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
[3:22:59 PM](0): Project generation failed.
[3:22:59 PM](4): Printing Bazel logs that could contain the error.
[3:22:59 PM](3): Log of Bazel aspect info output follows:
[3:22:59 PM](3): Bazel command info:
/opt/homebrew/bin/bazel build --announce_rc --show_result=0 --noshow_loading_progress --noshow_progress --symlink_prefix=/ --apple_platform_type=ios --cpu=ios_x86_64 --watchos_cpus=i386 --announce_rc '--override_repository=tulsi=/Users/diglesia/Library/Application Support/Tulsi/0.20221005.88/Bazel' --compilation_mode=dbg --define=apple.add_debugger_entitlement=1 --define=apple.propagate_embedded_extra_outputs=1 --define=apple.experimental.tree_artifact_outputs=1 --features=debug_prefix_map_pwd_is_dot --tool_tag=tulsi:generator --build_event_json_file=/var/folders/fm/yng8gs1s2kb1kwv6gvsv4nc4007656/T/build_events_65322.json --noexperimental_build_event_json_file_path_conversion --noexpand_test_suites --features=-parse_headers --noexperimental_run_validations --aspects @tulsi//:tulsi/tulsi_aspects.bzl%tulsi_sources_aspect --output_groups=tulsi_info,-dsyms //mediapipe/examples/ios/facedetectioncpu:FaceDetectionCpuApp //mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp //mediapipe/examples/ios/faceeffect:FaceEffectApp //mediapipe/examples/ios/facemeshgpu:FaceMeshGpuApp //mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp //mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp //mediapipe/examples/ios/helloworld:HelloWorldApp //mediapipe/examples/ios/holistictrackinggpu:HolisticTrackingGpuApp //mediapipe/examples/ios/iristrackinggpu:IrisTrackingGpuApp //mediapipe/examples/ios/objectdetectioncpu:ObjectDetectionCpuApp //mediapipe/examples/ios/objectdetectiongpu:ObjectDetectionGpuApp //mediapipe/examples/ios/objectdetectiontrackinggpu:ObjectDetectionTrackingGpuApp //mediapipe/examples/ios/posetrackinggpu:PoseTrackingGpuApp //mediapipe/examples/ios/selfiesegmentationgpu:SelfieSegmentationGpuApp //mediapipe/objc:mediapipe_framework_ios
Exited with code 1

INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /Users/diglesia/mediapipe/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /Users/diglesia/mediapipe/.bazelrc:
  'build' options: --jobs 128 --define=absl=1 --enable_platform_specific_config --apple_platform_type=macos --per_file_copt=.*.mm$@-std=c++17 --apple_generate_dsym
INFO: Found applicable config definition build:macos in file /Users/diglesia/mediapipe/.bazelrc: --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --copt=-w
DEBUG: Rule 'rules_foreign_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "c2cdcf55ffaf49366725639e45dedd449b8c3fe22b54e31625eb80ce3a240f1e"
DEBUG: Repository rules_foreign_cc instantiated at:
  /Users/diglesia/mediapipe/WORKSPACE:49:13: in <toplevel>
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/bazel_tools/tools/build_defs/repo/http.bzl:353:31: in <toplevel>
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/131dda9acc6978aba4740d75ca45f411fbabd726.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/8700849191dea2ca9c12a3d26ec41fac0814671b.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_google_absl' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_google_benchmark' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'flatbuffers' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'pybind11_bazel' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_googlesource_code_re2' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_google_protobuf' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_google_googletest' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_github_gflags_gflags' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'zlib' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'build_bazel_rules_apple' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'build_bazel_rules_swift' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'build_bazel_apple_support' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'xctestrunner' because it already exists.
DEBUG: /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'pybind11' because it already exists.
INFO: Repository local_execution_config_python instantiated at:
  /Users/diglesia/mediapipe/WORKSPACE:488:14: in <toplevel>
  /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/tensorflow/workspace2.bzl:879:19: in workspace
  /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/tensorflow/workspace2.bzl:86:27: in _tf_toolchains
  /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/tensorflow/tools/toolchains/remote_config/configs.bzl:6:28: in initialize_rbe_configs
  /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/tensorflow/tools/toolchains/remote_config/rbe_config.bzl:158:27: in _tensorflow_local_config
Repository rule local_python_configure defined at:
  /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/py/python_configure.bzl:279:41: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_execution_config_python':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/py/python_configure.bzl", line 216, column 39, in _create_local_python_repository
		numpy_include = _get_numpy_include(repository_ctx, python_bin) + "/numpy"
	File "/private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/py/python_configure.bzl", line 190, column 19, in _get_numpy_include
		return execute(
	File "/private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/remote_config/common.bzl", line 230, column 13, in execute
		fail(
Error in fail: Problem getting numpy include path.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
ERROR: /Users/diglesia/mediapipe/WORKSPACE:488:14: fetching local_python_configure rule //external:local_execution_config_python: Traceback (most recent call last):
	File "/private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/py/python_configure.bzl", line 216, column 39, in _create_local_python_repository
		numpy_include = _get_numpy_include(repository_ctx, python_bin) + "/numpy"
	File "/private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/py/python_configure.bzl", line 190, column 19, in _get_numpy_include
		return execute(
	File "/private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/org_tensorflow/third_party/remote_config/common.bzl", line 230, column 13, in execute
		fail(
Error in fail: Problem getting numpy include path.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
INFO: Repository ios_opencv instantiated at:
  /Users/diglesia/mediapipe/WORKSPACE:301:13: in <toplevel>
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_diglesia/2a7c7eef140c086170c425266ceff07c/external/bazel_tools/tools/build_defs/repo/http.bzl:353:31: in <toplevel>
ERROR: Analysis of target '//mediapipe/objc:mediapipe_framework_ios' failed; build aborted: Problem getting numpy include path.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
INFO: Elapsed time: 1.110s
INFO: 0 processes.
INFO: Build Event Protocol files produced successfully.

[3:22:59 PM](0): Generate[FAIL]: 1.3041s [Details]: General failure: Bazel aspects could not be built.
[3:22:59 PM](0): Failed to create Xcode project.

Hi @monkeyswarm,
Could you let us know if this closed issue #2701(Comment) resolves the above error. Thank you!

Yes, it does resolve it, thank you.
Is there a tracking issue for this kind of Tulsi environment variable issue?

Hi @monkeyswarm,
We do not have any this kind of tracking issue as of now but we see this can be brought up to notice of the internal team. Thank you!

Hi @monkeyswarm,
Could you update the status of the above issue if we can move this to close status as it has been resolved. Thank you!

source: gharchive/issue

authors: kuaashish monkeyswarm

license_type: permissive

license_source: github-api


text:
How to properly include a Licence Notice for using MediaPipe on a Webpage?

I'm using MediaPipes "Face Landmarker"-model in my webproject and would like to include the licence notice in the legal details.
I haven't found anything about it in the documentations or on the MediaPipe Solution webpage.
I'm also relatively new to webprogramming and don't realy know if i need the users consent before I use the "Face Landmarker"-model for the web.
Is there a chance anyone can gve me an example on how to include the licence in the legal details?
Kind regards!

Hi @Hybarbel,
Please allow us sometime to provide the support you are looking from us. Thank you!

We are using Apache License Version 2.0: https://github.com/google/mediapipe/blob/master/LICENSE
Please refer to this license text or any documentation you may find. We cannot give legal advise.

authors: Hybarbel kuaashish schmidt-sebastian


text:
dataset

Could you please share one dataset to train and render video? @keunhong

We'll release something soon. Please hang tight. Thanks.

Sorry for the delay. We've released out dataset.

authors: flynnamy keunhong

repo: google/nerfies

license_source: bigquery


text:
Running in Docker

Hi, any idea how to get this running inside a docker container? Currently integrated as part of WebGL unit testing which works perfectly. Except, fails on the CI/CD side since this is a docker environment. Makes sense since there are no GPU drivers. But even emulation would be acceptable since performance is a non-issue. This is all under Ubuntu. Any help is much appreciated.

Solved by installing xvfb (apt) and xvfb-maybe (npm). Then as part of package.json "test": "xvfb-maybe jest"

repo: google/node-gles


text:
Floating point exception error in CP-SAT solver

What version of OR-Tools and what language are you using?
Version: v9.9.3903
Language: Python
Which solver are you using (e.g. CP-SAT, Routing Solver, GLOP, BOP, Gurobi)
CP-SAT
What operating system (Linux, Windows, ...) and version?
Ubuntu 22.04 LTS (WSL2 on Windows 11)
What did you do?
Run this model:
test_model.txt
What did you expect to see
Solution
What did you see instead?
Floating point exception error
Make sure you include information that can help us debug (full error message, model Proto).
test_log.txt
Anything else we should know about your project / environment
NA

Thanks. I will have a look

Also, another thing is that "Floating point exception" error is not printed in Windows. It just ends abruptly.

Thanks a lot.

authors: arnabadas lperron

repo: google/or-tools


text:
Add Track::IsFilterable() method to allow non-filterable tracks

Scheduler track is now always visible (b/181671054).
"Pinned" tracks are also not be filterable.

but this will break some E2E tests

Arrgh, I will implement your suggestions, thanks @reichlfl

source: gharchive/pull-request

repo: google/orbit


text:
Generalize MapInfo to use Object instead of Elf and rename appropriately

In order to use different types of object files we have already
introduced an more general Object class. This class is now also
used in MapInfo, which is generalized to general object file types.
This unfortunately triggers a large number of required renaming and
changes throughout the library.

The main changes are in MapInfo and Unwinder, using Object and
renaming a couple of fields from "elf" to "object" to generalize.
The Elf cache is moved up to the Object class. This is mainly
moving code from Elf.cpp to Object.cpp. This could probably be its own
change, but it felt inconsistent without this step.
Remaining changes (in tests and Regs* classes) are just changes
triggered by renaming.

Tested: Unit tests; built version and ran on a sample application.
Bug: http://b/192514457

@danielfenner, @beckerhe Apologies for the rather large change. Please let me know if I should try to separate out the Elf cache to Object cache change.


text:
Is it possible to bypass the --full-sort option if we assign timestamps in MACROS

Hi,
First, thanks for the great help all the time!
Problem description
In our case, the trace file is post generated rather using a hook based approach. Which means we pass timestamps to the MACROS, e.g. TRACE_COUNTER(counter_category, counter_track, timestamp, val);.
Currently, I use set_write_into_file option. But because of we're not using a hook based approach, options like set_file_write_period_ms does not work. The events are not fully sorted. Inside each track's data, the events are sorted. But when we generate all the tracks together, we go track by track which makes the events not sorted.
I'm flushing the trace manually with tracing_session->FlushBlocking(flush_timeout); every couple thousands TRACE_EVENT.
I'm getting this sorting error below if not passing the --full-sort option.
                                    name                                      idx                                   source                                    value
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------------------------------
counter_events_out_of_order              [NULL]                                   analysis                                                                   393689
slice_out_of_order                       [NULL]                                   analysis                                                                  9178010
sorter_push_event_out_of_order           [NULL]                                   trace                                                                     9571771

Reason
The reason we want to avoid using --full-sort with trace_processor is it take a lot of memory.
FMHO, I don't know why they need to be sorted. Shouldn't the Sqlite3 database index the timestamp and kind of sort events when inserting them into the database?
Possible solution
One method I can think of is using priority queue and merge K sorted array. But that will change our API completely.
Am I correct with that --full-sort option? Can I bypass that option with other method?
Context
I'm under perfetto v24.2 branch.
Thanks,

Not sure what you mean with " hook based approach" but I guess you want to control the timing of reads.
Note that Flush is not enough. Flush just causes producer processes to commit all the data in the ring buffer, but does not cause the ring buffer to be written into the file. That takes an internal ReadBuffers call.
I don't know how you would achieve that in your case though, if you set set_write_into_file but then don't want to use set_file_write_period_ms (why? what's the rationale for that) you don't have other ways to control when that should happen.
Issuing a ReadBuffer IPC won't work, because that is designed to return data via IPC, but if the trace is write_into_file=true, that will refuse to work, because you can't do IPC reads on a trace that is setup to write into a file.

FMHO, I don't know why they need to be sorted.

Because a lot of code and optimizations internally assume table data is sorted to avoid doing post-facto sorting at query time which would be expensive.

Shouldn't the Sqlite3 database index the timestamp and kind of sort events when inserting them into the database?

No, for many reasons:

We don't use the standard sqlite in-memory tables. Those are too expensive/slow, they as are tuple based. We have our own columnar-oriented in memory-storage, which allows us to have no indices at all (we create indexes on the flight base on columns).
But then some columns (e.g. ts) are assumed to be sorted, because re-sorting them in a large trace is too expensve and requires reshuffling all the data. We will never go there, there is no reason to go there.
It's not just about SQL queries. A lot of events need to be seen in order by the importer, as often some events (E.g. clock snapshots) are requires to interpret/add semantic to events that come after. If we didn't rely on this we'd have to do multi-pass imports on the trace but that's too much.
If it takes too much memory, enable swap. that's what swap is for literally.

hook based approach

This refers to how the events are triggered. Since you're using the MACROS every time it's triggered, it uses the current time. This behaves like a hook.

I don't know how you would achieve that in your case though, if you set set_write_into_file but then don't want to use set_file_write_period_ms (why? what's the rationale for that) you don't have other ways to control when that should happen.

I set all the options like this. But it turns out I still need to use --full-sort.
  cfg.set_write_into_file(true);
  cfg.set_file_write_period_ms((uint32_t)1000);
  cfg.set_flush_period_ms((uint32_t)1000);

I assume that period_ms is based on the current time. If we manually pass timestamps to the TRACE_EVENT, e.g.TRACE_COUNTER(counter_category, counter_track, timestamp, val);. Then that file_write_period_ms won't work as we expected.
I created a minimal reproducible example here. This explains our use case.

Without tracing_session->FlushBlocking(flush_timeout), it gives metraced_buf_trace_writer_packet_loss .
With FlushBlocking, I can get a track like the image below. But I'll get counter_events_out_of_order, sorter_push_event_out_of_order if --full-sort is not passed.

As file_write_period_ms does not work as we expected, what API should I call to manually do internal ReadBuffers call?

The simple fact of the matter is that you are using the concept of "Flush" in a way which is orthogonal to how Perfetto expects it.
A Flush in Perfetto terms = push all data which was emitted before the flush timestamps to the central buffers. If you're faking events, this essentially means when you flush, you should "never* emit timestamps before the max timestamp before the flush.
All the trace processor incremental sorting logic depends on this invariant to hold: otherwise we have no choice but to sort the trace. No matter how you call the APIs or what you do, if you don't respect this invariant, we cannot incrementally sort.
One thing I am working is reducing the amount of memory a full sort takes as we're hitting limits on this ourselves for traces where we do not use write_into_file at all. This is ideal for a new starter so I won't be doing the work myself but I'll oversee it when possible.
As of now, I don't think there's anything we can do in this bug so I'm going to close it.

Thanks for the clarification.
One way on our end is trying merging K sorted lists. Because inside each track, the events are sorted.
👍

authors: LalitMaganti primiano ztlevi

repo: google/perfetto


text:
Include in postcss-preset-env ?

Should this polyfill be included in postcss-preset-env?
From what I can tell it seems relatively safe for users because it throws when something unsafe is encountered.

I think that would be reasonable, but it's really a decision for the postcss-preset-env owners.

I am one of the owners.
Let me rephrase.

I think this plugin would be a nice addition to postcss-preset-env and I would like to add it to postcss-preset-env. I don't want to add it unless you agree it's a good candidate.

Oh, I see! Yes, I think it would :smile:

Nice! 😄
I've drafted a PR : https://github.com/csstools/postcss-plugins/pull/1021
I will let you know when this ships 🙇

repo: google/postcss-src


text:
Anyone seeing this asObjCProimse swiftc crash in Xcode 11

for this method:
  public func allFoo() ->
    Promise<NSArray>.ObjCPromise<NSArray>
  {
    return promiseSwiftArrayOfFoo().then { $0 as NSArray }.asObjCPromise()
  }

Crashes swiftc with:
Showing All Messages
1.	While generating Objective-C header [redacted]
2.	While printing 'TheClass' at [redacted]/TheClass.swift:14:8
3.	While printing type 'Promise<NSArray>.ObjCPromise<NSArray>'
4.	While printing type 'Value'

Interesting, a reduced example doesn't cause the compiler crash. I was thinking it might be getting confused by the re-use of Value, but in simpler cases it doesn't seem to be the case.

Hi Mark,
What if you try something like:
public func allFoo() -> Promise<[Foo]>.ObjCPromise<NSArray> {
  return promiseSwiftArrayOfFoo().asObjCPromise()
}

That was actually what I was doing - I simplified it to use the map in case that helped, both crash in my code.
What does seem to work though is explicitly using FBLPromise<NSArray> as the return type. Compiles fine, haven't tested it yet.

Ok, so a current workaround is:
import Promises
import FBLPromise

  @objc
  public func allFoo() -> FBLPromise<NSArray>
  {
    return promiseSwiftArrayOfFoo().asObjCPromise()
  }

Despite requiring the extra import, this actually reads a bit cleaner, so I'm ok with it. Of course swiftc shouldn't crash, and I'll keep trying to make a reduced sample that does trigger the crash.

Interesting... So you mean if you replace Promise<[Foo]>.ObjCPromise<NSArray> return type with FBLPromise<NSArray>, it does fix the crash?

Yes.

The crash is in the name mangling/compression code when swiftc is generating the ObjC header, so it sort of makes sense. When the type is a plain ObjC type that code doesn't need to run, it can just use the ObjC type directly.

Would you mind to make a small text case demonstrating the issue?
May just edit one of the test files we already have, like Promise+ThenInteroperabilityTests.swift, or  create a custom one.
Happy to play with it and see if we can do better.

Hey Mark, going to close this issue for now. Feel free to reopen if you have any additional details or can provide a simple test case that repros the issue. Thank you!

authors: aufflick shoumikhin temrich

repo: google/promises


text:
Fix various regressions from current Android binaries

Bit 9 in OS/2's fsSelection field is set for italics, since table version became 4 with the use of ufo2ft
Unique ID and version in name table now conform to OT spec
head table flags and lowestRecPPEM are back to old values
cmap format 4 is dropped in Android post-production, to save space

LGTM

authors: dougfelt jamesgk

repo: google/roboto


text:
Included idris lawal technical writer

I have included my name in the technical writers section.

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
Please visit https://cla.developers.google.com/ to sign.
Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.
What to do if you already signed the CLA
Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

Googlers: Go here for more info.

@googlebot I signed it!

repo: google/season-of-docs


text:
Implement Pooling/Convolutional Encoder

[ ] Implement the convolutional encoder described in https://arxiv.org/abs/1611.02344.
[x] Implement the baseline pooling encoder with position embeddings from the same paper.

Done in #100


text:
open-source test doc update (MPW5)

Signed-off-by: Mehdi Saligane mehdi.saligane@gmail.com
Fixes #6

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[x] Appropriate changes to README are included in PR

We should make sure to keep the proprietary test tile and the new open source test tile documentation seperate from each other.

authors: mithro msaligane


text:
fallback to .init.text or .init.plt section address in case .text not exist or 0 sized

Current KernelModule is generated from .text section for modules,
for some module, there is no .text section.
This results in size field in KernelModule set to 0 when we check
/modules?raw=true.
When we use the modules info to decide which module the pc belongs in
findModule function, it might gives wrong info.
To make the findModule work properly, we need pc from .init.text
discard correctly and pc from .text find correct module.

Before sending a pull request, please review Contribution Guidelines:
https://github.com/google/syzkaller/blob/master/docs/contributing.md

Why is this needed/useful? We never fuzz/cover .init.text section.

add commit msg and updated the same in the PR description.

Why is this needed/useful? We never fuzz/cover .init.text section.

add commit msg and updated the same in the PR description.

I still don't understand.
.init.text is unloaded after module loading, so never cover/fuzz it.
Why does findModule return wrong module in such cases? It knows that the module text size is 0, so it should never return such modules. Is there a bug in findModule that we need to fix?

About ".init.text is unloaded after module loading, so never cover/fuzz it.", I feel confused.
On my device, I have
# zcat /proc/config.gz |grep UNLOAD
CONFIG_MODULE_UNLOAD=y

But I can see .init.text section like
# cat /sys/module/zram/sections/.init.text
0xffffffe15601009c

Does it mean .init.text is not unloaded?
The test show the current code is fine. So it's my local extra script hasn't been updated to use findModule logic.
func TestFindModule(t *testing.T) {
	moduleAddresses := []uint64{0x00015000, 0x00020000, 0x00030000, 0x00040000, 0x00045000}
	moduleSizes := []uint64{0x5000, 0x5000, 0x10000, 0x5000, 0x10000}
	modules := initModules(moduleAddresses, moduleSizes)
	moduleKeys := make([]uint64, len(modules))
	setModuleKeys(moduleKeys, modules)
	pc := uint64(0x00017000)
	moduleIdx := findModule(pc, moduleKeys)
	assert.Equal(t, moduleIdx, 0)

	moduleAddresses = []uint64{0x00015000, 0x00020000, 0x00030000, 0x00040000, 0x00045000}
	moduleSizes = []uint64{0x0, 0x0, 0x10000, 0x5000, 0x10000}
	modules = initModules(moduleAddresses, moduleSizes)
	moduleKeys = make([]uint64, len(modules))
	setModuleKeys(moduleKeys, modules)
	pc = uint64(0x00021000)
	moduleIdx = findModule(pc, moduleKeys)
	assert.Equal(t, moduleIdx, 1)
}

About ".init.text is unloaded after module loading, so never cover/fuzz it.", I feel confused.

Ok, maybe it's not unloaded, but it shouldn't be fuzzed since it contains only module initialization code, right.
And that's what matters in this context.

So this PR should not be uploaded since it's not for __sanitizer_cov_trace_pc from .text section.

Should we close the PR then?

Thanks for your explanation on unload part, yes, closed the PR.

@dvyukov There is one more question, since findModule is using index only, there is no need to use Size field in KernelModule, right?
func (linux) parseModules(files filesystem) ([]*KernelModule, error) {
	var modules []*KernelModule
	re := regexp.MustCompile(`(\w+) ([0-9]+) .*(0[x|X][a-fA-F0-9]+)[^\n]*`)
	modulesText, _ := files.ReadFile("/proc/modules")
	for _, match := range re.FindAllSubmatch(modulesText, -1) {
		name := string(match[1])
		modAddr, err := strconv.ParseUint(string(match[3]), 0, 64)
		if err != nil {
			// /proc/modules is broken, bail out.
			return nil, fmt.Errorf("module %v address parsing error: %w", name, err)
		}
		textAddr, err := linuxModuleTextAddr(files, name)
		if err != nil {
			// Module address unavailable, .text is probably 0. Skip this module.
			continue
		}
		modSize, err := strconv.ParseUint(string(match[2]), 0, 64)
		if err != nil {
			// /proc/modules is broken, bail out.
			return nil, fmt.Errorf("module %v size parsing error: %w", name, err)
		}
		offset := modAddr - textAddr
		modules = append(modules, &KernelModule{
			Name: name,
			Addr: textAddr,
			// The size is wrong as there is overlap in /proc/modules
			// ex. module1['Addr'] + module1['Size'] > module2['Addr']
			// runtime kernel doesn't export .text section size to /sys/module/*/sections/.text
			// so we need to read it from elf
			Size: modSize - offset,
		})
	}

I think we don't need to get Size here, should we remove it?
And in the case, module.Addr is 0 for these modules which don't have .text section, right?
func linuxModuleTextAddr(files filesystem, module string) (uint64, error) {
	data, err := files.ReadFile("/sys/module/" + module + "/sections/.text")
	if err != nil {
		return 0, fmt.Errorf("could not read module %v .text address file: %w", module, err)
	}
	addrString := strings.TrimSpace(string(data))
	addr, err := strconv.ParseUint(addrString, 0, 64)
	if err != nil {
		return 0, fmt.Errorf("address parsing error in %v: %w", module, err)
	}
	return addr, nil
}

What's the original problem you are trying to solve? Is something crashing or misbehaving?
I would keep size for sanitizer checking, findModule could check that the requested PC fits within [addr, addr+size]. Addr without size looks incomplete. Without size we can only assume that the module takes all space up to the next module, but this can be very large range (for the last module and between vmlinux and modules). For example, we can miss some dynamically loaded module and then start attributing coverage from it to a random preceding module.
If we have a module for which we inferred 0 text size, we could not keep it at all and not consider in findModule. Would this solve your problem?

authors: dvyukov jiangenj

repo: google/syzkaller


text:
Enable graph checking by default

Also:

Fix dtype in numpy feature class.

Thanks

repo: google/temporian


text:
Put pb.go files in repo

It would be nice if tink put the pb.go files in the repo. This would allow Go projects to use Tink without also requiring them to also become Basil projects.

Cool. Will find a way to do this.

See the trillian repo for an example.

Hmm I don't really know how to do this. Gary if you know how, could you please send a PR?

This can be closed now?

authors: gdbelvin thaidn

repo: google/tink


text:
Unable to retrieve logical volume

PartitionEnumerationTask Task failed with exception: [pyvslvm_volume_group_get_logical_volume_by_index: unable to retrieve logical volume: 0. libvslvm_logical_volume_initialize: unsupported number of stripes. libvslvm_volume_group_get_logical_volume: unable to create logical volume.]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/turbinia-20210208.4.dev19+g7c02e5e-py3.6.egg/turbinia/workers/__init__.py", line 893, in run_wrapper
    self.result = self.run(evidence, self.result)
  File "/usr/local/lib/python3.6/dist-packages/turbinia-20210208.4.dev19+g7c02e5e-py3.6.egg/turbinia/workers/partitions.py", line 144, in run
    path_specs = partitions.Enumerate(evidence)
  File "/usr/local/lib/python3.6/dist-packages/turbinia-20210208.4.dev19+g7c02e5e-py3.6.egg/turbinia/processors/partitions.py", line 47, in Enumerate
    path_specs = scanner.GetBasePathSpecs(evidence.local_path)
  File "/usr/local/lib/python3.6/dist-packages/dfvfs/helpers/volume_scanner.py", line 594, in GetBasePathSpecs
    self._source_scanner.Scan(scan_context)
  File "/usr/local/lib/python3.6/dist-packages/dfvfs/helpers/source_scanner.py", line 661, in Scan
    self._ScanNode(scan_context, scan_node, auto_recurse=auto_recurse)
  File "/usr/local/lib/python3.6/dist-packages/dfvfs/helpers/source_scanner.py", line 479, in _ScanNode
    scan_context, scan_node, auto_recurse=auto_recurse)
  File "/usr/local/lib/python3.6/dist-packages/dfvfs/helpers/source_scanner.py", line 606, in _ScanVolumeSystemRootNode
    for sub_file_entry in file_entry.sub_file_entries:
  File "/usr/local/lib/python3.6/dist-packages/dfvfs/vfs/lvm_file_entry.py", line 101, in _GetSubFileEntries
    yield LVMFileEntry(self._resolver_context, self._file_system, path_spec)
  File "/usr/local/lib/python3.6/dist-packages/dfvfs/vfs/lvm_file_entry.py", line 65, in __init__
    path_spec)
  File "/usr/local/lib/python3.6/dist-packages/dfvfs/vfs/lvm_file_system.py", line 133, in GetLVMLogicalVolumeByPathSpec
    return self._vslvm_volume_group.get_logical_volume(volume_index)
OSError: pyvslvm_volume_group_get_logical_volume_by_index: unable to retrieve logical volume: 0. libvslvm_logical_volume_initialize: unsupported number of stripes. libvslvm_volume_group_get_logical_volume: unable to create logical volume.

No state_manager initialized, not updating Task info
Trying last ditch attempt to close result
Task Result was auto-closed from task executor on turbinia-worker-6eaf93e7aee7c9cf-0 likely due to previous failures.  Previous status: [PartitionEnumerationTask Task failed with exception: [pyvslvm_volume_group_get_logical_volume_by_index: unable to retrieve logical volume: 0. libvslvm_logical_volume_initialize: unsupported number of stripes. libvslvm_volume_group_get_logical_volume: unable to create logical volume.]]

FYI @dfjxs

libvslvm (https://github.com/libyal/libvslvm) currently only supports simple LVM volumes.
This image used for testing is not supported.

Is this something that we can determine at enumeration time and issue some kind of warning then?

I think we can close this out and rely on the upstream bug to fix this.

authors: aarontp dfjxs

repo: google/turbinia


text:
Display Rules for Links

Link tooltip should be visible only on hover.
Expected Behaviour
Link tooltip appears on hover (not when the element is selected).
Additional Context
From UX Evaluation

Plugin Version:
Operating System:
Browser:

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

The 2-tap UI (old popup design for now) appears on hover (not when the element is selected Figma )

Implementation Brief

@samitron7 to add details regarding what happens if the element is rotated.

Issue with blobs with weird sizes and at weird rotations:

Just chatted with Omar and we decided to not show the link icon. I made changes to the UI to reflect removal of the link icon. Figma
@jauyong Can we ensure that we change the title and AC to the ticket or close this one and open a new one for the following:

When a link is applied to the element, we show the 2-tap UI as confirmation. The confirmation should fade away. Figma
The 2-tap ui appears on hover (not when the element is selected Figma
Lastly, the fields for the metadata (ie description and brand icon) should only appear if the user enters a valid url. Currently, if you enter in garbage in the web address, the metadata fields are available.

Removing UX needed label

@merapi, please update the issue description to reflect @samitron7's comment on what is actually to be implemented here (as both description and title is currently completely misleading).

For clarification, since @merapi and @samitron7 discussed offline, it is ok to use the standard from the format for this Beta release and delay additional UI changes to post Beta.

Dropping in a screenshot of the conversation:

@merapi We're good to go but @barklund and @jauyong this would mean we will need to file another ticket. I left another comment above and copying here again so it doesn't get lost:
@jauyong Can we ensure that we change the title and AC to the ticket or close this one and open a new one for the following:
When a link is applied to the element, we show the 2-tap UI as confirmation. The confirmation should fade away. Figma
The 2-tap ui appears on hover (not when the element is selected Figma
Lastly, the fields for the metadata (ie description and brand icon) should only appear if the user enters a valid url. Currently, if you enter in garbage in the web address, the metadata fields are available.

Hey sorry for the delay in getting to this, I see it's already in review but I'll add the AC for Review / QA purposes

Lastly, the fields for the metadata (ie description and brand icon) should only appear if the user enters a valid url. Currently, if you enter in garbage in the web address, the metadata fields are available.

I don't think this groups well with the other two ACs, nor dos #2460 cover this. Instead, I'll cover this as part of #2254.

Verified in QA - PR#2460

Verified in UAT

authors: barklund csossi jauyong o-fernandez samitron7 swissspidy winterDroid

repo: google/web-stories-wp


text:
Taxonomy: Set feature flag to default true

Acceptance Criteria
Set taxonomy feature flag to default: true.
Add QA directions as this issue will act as QA of the taxonomy feature.

Verified in QA

Verified in QA

authors: BrittanyIRL csossi


text:
%YAML:1.0 give error

when in the start of the file there is %YAML:1.0
then tells that yaml including non alphabetical character and not linting file.

Hi @niroosh1997 thanks for opening an issue.
It looks like the underlying yaml library does not support directives. I could potentially add a hotfix where yamlfmt purposely strips directives and puts them back in after the document similar to other features I've added. Worth noting though, the YAML 1.0 spec is similarly not supported by the YAML parser (see that library's README).

Seems like great hotfix for my purposes!
I did it manually and it works great so would love this hotix
Thanks Nir
On Wed, 30 Oct 2024, 20:48 Braydon Kains, @.***> wrote:

Hi @niroosh1997 https://github.com/niroosh1997 thanks for opening an
issue.
It looks like the underlying yaml library does not support directives. I
could potentially add a hotfix where yamlfmt purposely strips directives
and puts them back in after the document similar to other features I've
added. Worth noting though, the YAML 1.0 spec is similarly not supported by
the YAML parser (see that library's README
https://github.com/go-yaml/yaml?tab=readme-ov-file#compatibility).
—
Reply to this email directly, view it on GitHub
https://github.com/google/yamlfmt/issues/215#issuecomment-2448078675,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AME6ZHCCVWPP4Y2EBFXGTTLZ6ESX3AVCNFSM6AAAAABQ4KBOKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBYGA3TQNRXGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

I will try and get something added in soon, along with documentation stating that directives are ignored and will have no affect on parsing.

repo: google/yamlfmt


text:
ci: use ubuntu-latest runner for flutter job

Description
Change as per discussion here: https://github.com/googleads/googleads-mobile-flutter/pull/223#discussion_r632821963
cc @blasten
Related Issues
N/A
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process. Updating the pubspec.yaml and changelogs is not required.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

Seems a few minutes faster compared to previous runs:

We might also be able to use Ubuntu for the Android job.

We might also be able to use Ubuntu for the Android job.

@blasten Yep I think you're right, would only need to use the macos runner if using an Android emulator - but in this case it's just build & unit tests. Want me to change it over too?

sgtm

@Salakar do you still want to update the OS ?   The Android emulators also run on Ubuntu.

authors: Salakar blasten

repo: googleads/googleads-mobile-flutter


text:
Installation is failed with pipsi

Hi,
I'm trying to install artman with pipsi.
I clone the code and run pipsi install . within directory. But failed with these logs:
[realityone@rEimu ~/Softs/artman]$ pipsi install .
Traceback (most recent call last):
  File "/Users/realityone/.local/bin/pipsi", line 11, in <module>
    sys.exit(cli())
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/click/decorators.py", line 27, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/pipsi.py", line 366, in install
    if repo.install(package, python, editable):
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/pipsi.py", line 215, in install
    venv_path = self.get_package_path(package)
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/pipsi.py", line 187, in get_package_path
    return join(self.home, normalize_package(package))
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/pipsi.py", line 61, in normalize_package
    requirement = Requirement.parse(value)
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3009, in parse
    req, = parse_requirements(s)
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2956, in parse_requirements
    yield Requirement(line)
  File "/Users/realityone/.local/venvs/pipsi/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2965, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "': 0.15.7'"

After digging into the setup.py file, I found we should remove the
print('Version: %s' % current_version)

line, otherwise pipsi will failed to parse requirements.
Is this add by mistake or this is by designed.
Any help will be appreciated.

Hello @realityone,
You're right, looks like pipsi install . is broken. I'm fixing it in #512 and will release the new version when it's merged. Until then, you can just remove that print from setup.py.
Thanks!

authors: alexander-fenster realityone

repo: googleapis/artman


text:
updated service REGEX to accept additional valid service name values

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

[x] Make sure to open an issue as a bug/issue before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
[x] Ensure the tests and linter pass
[x] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate docs were updated (if necessary)

Fixes #: https://console.cloud.google.com/support/cases/detail/29452002

We're updating the backend part of this now. Once that's ready, this PR can be reviewed.
Meanwhile, the CLA is missing, and the commit message isn't following conventional commits.
Can you sign the CLA and update the commit message?

Looks like there's no activity here.
We merged a PR that's doing the same thing, so this PR is subsumed by that one.

repo: googleapis/cloud-profiler-nodejs


text:
Feature request: Extract and propagate B3 headers

Istio supports distributed tracing on an ingress into a cluster, and between services.
To do this, it propagates a standard set of B3 headers (originally defined for Zipkin, I think?)
x-request-id
x-b3-traceid
x-b3-spanid
x-b3-parentspanid
x-b3-sampled
x-b3-flags
x-ot-span-context

It would be great if the server side portions of this tracer recognised and propagated the information from that source. opencensus-node apparently supports this, though it is itself in very early alpha, and has no GRPC support.
I can share simple istio config that defines this behaviour, if it's helpful.

Marking this blocked because #950 is blocked.

authors: majelbstoat ofrobots

repo: googleapis/cloud-trace-nodejs


text:
google-api-core distribution not found

Hi,
I have installed the library using
pip install --upgrade google-api-python-client

Into my virtual env. Everything is working just fine. Although when I push to production and have my build server install packages from the requirements.txt file, I get the error.
The 'google-api-core' distribution was not found and is required by the application

Although, I can see that the package has been installed (or so the build logs say).
Production Env: Using Seed + Serverless
Any help would be great!

Can you give us the contents of your requirements.txt?
Can you give more details on how you deploy to the production environment? This sort of issue can happen when pkg_resources doesn't have access to package metadata, which can be caused by strange environments that don't use a real python (virtual) environment, like GAE standard.

Hi Jon
So I am using "Serverless" (AWS Lamba) - although for the CI infrastructure/tool I am using "Seed". Below is my requirements.txt
aniso8601==1.3.0
apispec==0.25.4
appnope==0.1.0
awscli==1.11.146
backports.shutil-get-terminal-size==1.0.0
boto3==1.4.8
botocore==1.8.1
cachetools==2.0.1
certifi==2018.1.18
cffi==1.11.2
chardet==3.0.4
click==6.7
colorama==0.3.7
decorator==4.1.2
docutils==0.14
enum34==1.1.6
ez-setup==0.9
facebook-sdk==2.0.0
flasgger==0.8.0
Flask==0.12.2
flask-apispec==0.4.0
Flask-Cors==3.0.3
flask-restplus==0.10.1
functools32==3.2.3.post2
futures==3.2.0
gitdb2==2.0.3
GitPython==2.1.7
google-api-core==0.1.4
google-api-python-client==1.6.5
google-auth==1.3.0
google-cloud-core==0.28.0
google-cloud-language==1.0.0
googleapis-common-protos==1.5.3
grpcio==1.9.1
gunicorn==19.7.1
httplib2==0.10.3
idna==2.6
ipython==5.4.1
ipython-genutils==0.2.0
itsdangerous==0.24
Jinja2==2.10
jmespath==0.9.3
jsonschema==2.6.0
MarkupSafe==1.0
marshmallow==2.13.6
mistune==0.8.1
mongoengine==0.13.0
msgpack==0.5.0
msgpack-python==0.4.8
numpy==1.14.0
oauth2client==4.1.2
pathlib2==2.3.0
pexpect==4.2.1
pickleshare==0.7.4
prompt-toolkit==1.0.15
protobuf==3.5.1
ptyprocess==0.5.2
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.18
Pygments==2.2.0
pymongo==3.5.1
python-dateutil==2.6.1
python-slugify==1.2.4
pytz==2017.3
PyYAML==3.12
requests==2.18.4
rsa==3.4.2
s3transfer==0.1.11
scandir==1.5
simplegeneric==0.8.1
six==1.11.0
smmap2==2.0.3
traitlets==4.3.2
Unidecode==0.4.21
uritemplate==3.0.0
urllib3==1.22
virtualenv==15.1.0
wcwidth==0.1.7
webargs==1.8.1
Werkzeug==0.12.2

Can you share anything about how your application is packaged up for deployment?

Getting the same error.
I'm getting this project as dependency of another one (dialogflow) but I can reproduce the problem.
I'm bootstrapping my dependencies into a specific dir so lets replicate that
$ pip install --prefix ./tmp google-api-core

This will now throw an exception:
import site
import pkg_resources

site.addsitedir('./tmp/lib/python3.6/site-packages')
pkg_resources.get_distribution('google-api-core')

This however works:
import site
site.addsitedir('./tmp/lib/python3.6/site-packages')

import pkg_resources
pkg_resources.get_distribution('google-api-core')

Seems like the import order of pkg_resources is important which I can not control in my case.
Any suggestions?

I was having the same issue.  After upgrading pip via pip install --upgrade pip, the error went away.

I'm going to close this issue as @crwilcox seems to have identified the problem and provided a workaround.  If this is in error, please let me know, and let me know what more is required.  Thanks!

I am with the same problem, I am with serverless in the last version and using the range of versions.

Sourced form serverless #6875
Description

What did you do?

I run pip install -r requirements.txt and all the imports work. I've done this in a isolated virtual environment running Python 3.7. Running sls deploy is fine.
sls invoke -f stagger --log fails because of a missing dependency.
my requirements.txt
boto3==1.10.0
fasttext==0.9.1
google-api-python-client==1.7.11
google-cloud-translate==1.7.0
spacy==2.2.1
Flask==1.1.1
joblib==0.14.0
scikit-learn==0.21.3
https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz

What happened?

{
    "errorMessage": "The 'google-cloud-translate' distribution was not found and is required by the application",
    "errorType": "DistributionNotFound",
    "stackTrace": [
        "  File \"/var/lang/lib/python3.7/imp.py\", line 234, in load_module\n    return load_source(name, filename, file)\n",
        "  File \"/var/lang/lib/python3.7/imp.py\", line 171, in load_source\n    module = _load(spec)\n",
        "  File \"<frozen importlib._bootstrap>\", line 696, in _load\n",
        "  File \"<frozen importlib._bootstrap>\", line 677, in _load_unlocked\n",
        "  File \"<frozen importlib._bootstrap_external>\", line 728, in exec_module\n",
        "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
        "  File \"/var/task/handler.py\", line 14, in <module>\n    from translate import detect_and_translate\n",
        "  File \"/var/task/translate.py\", line 5, in <module>\n    from google.cloud import translate\n",
        "  File \"/tmp/sls-py-req/google/cloud/translate.py\", line 18, in <module>\n    from google.cloud.translate_v2 import __version__\n",
        "  File \"/tmp/sls-py-req/google/cloud/translate_v2/__init__.py\", line 20, in <module>\n    __version__ = get_distribution(\"google-cloud-translate\").version\n",
        "  File \"/tmp/sls-py-req/pkg_resources/__init__.py\", line 481, in get_distribution\n    dist = get_provider(dist)\n",
        "  File \"/tmp/sls-py-req/pkg_resources/__init__.py\", line 357, in get_provider\n    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]\n",
        "  File \"/tmp/sls-py-req/pkg_resources/__init__.py\", line 900, in require\n    needed = self.resolve(parse_requirements(requirements))\n",
        "  File \"/tmp/sls-py-req/pkg_resources/__init__.py\", line 786, in resolve\n    raise DistributionNotFound(req, requirers)\n"
    ]
}

What should've happened?

Should call a Lambda function on AWS.

What's the content of your serverless.yml file?

service: stagger

plugins:
  - serverless-python-requirements
  - serverless-wsgi
  - serverless-offline
custom:
  wsgi:
    app: app.app
    packRequirements: false
  pythonRequirements:
    dockerizePip: non-linux
    zip: true

provider:
  name: aws
  runtime: python3.7
  timeout: 29
  stage: dev
  region: eu-west-1
  apiKeys:
      - myStaggerKey

functions:
  stagger:
    handler: handler.stagger
    events:
      - http:
          path: stagger
          method: post
          private: true

package:
    exclude:
        - venv/**

What's the output you get when you use the SLS_DEBUG=* environment variable (e.g. SLS_DEBUG=* serverless deploy)

No errors.
Similar or dependent issues:

#5026
Issue on googleapis

@tomonodes I think the responder on the original issue recommend you file the issue in https://github.com/serverless/serverless-google-cloudfunctions.

Looks like this issue is related to the Google Cloud Functions provider plugin. Could you re-open in that repository since serverless/serverless only covers the core and the AWS implementation. Thanks!

When you file issues on this repo in the future, please open a new issue rather than posting on a closed issue. Maintainers are more likely to miss posts to closed issues. Thanks!

authors: Tadly busunkim96 jonparrott luisdemarchi mcdonc prakashraman soumak77 tomonodes

repo: googleapis/google-api-python-client


text:
Proposal: consistent names for libraries and targets in CMake

Status
This has been implemented. We are waiting until 2022-02-15 to retire the legacy Bazel rules, CMake packages, and CMake rules.
Background
The Bigtable client library is called libbigtable_client.{a,so}, the CMake target is called bigtable_client.  The storage library follows the same pattern.
These names are prone to conflicts. Any other application or library provider could create a storage_client target.
When it came time to write Spanner we created an alias googleapis-c++::spanner_client, which at least is less prone to conflicts, but it is too redundant: it is unlikely that our Java googleapis will have CMake targets.  PubSub also uses the googleapis-c++:: prefix.
The common libraries (libgoogle_cloud_cpp_common.{a,so} and libgoogle_cloud_cpp_grpc_utils.{a,so}) do not have targets with the googleapis-c++:: prefix.
The proto libraries are exported with the googleapis-c++:: prefix.
Proposal

For CMake we will use a prefix (google-cloud-cpp::) consistently across all our exported libraries.
For Bazel we will recommend that applications use //:bigtable, //:pubsub, etc. as the target they link.
For CMake we will recommend google-cloud-cpp::bigtable, google-cloud-cpp::storage
For CMake packages will be imported with:find_package(google_cloud_cpp_bigtable)
find_package(google_cloud_cpp_pubsub)
find_package(google_cloud_cpp_storage)

Though rarely used, proto libraries will be imported with:find_package(google_cloud_cpp_googleapis)

T: As Soon as Possible

Export all libraries with the google-cloud-cpp:: prefix
Change all our examples and tests to use this prefix
Add aliases for the old names (storage_client, bigtable_client, etc.)
Declare these aliases as deprecated and announce that they will be removed ~in 2 * X months~ by a certain date.
Include this bug as a reference
We should explore using the DEPRECATION property to warn users.
Change the Bazel in google/cloud/*/BUILD to use the deprecation directive and reference this bug (as well as the top-level targets), see #3701
Rename the physical libraries to libgoogle_cloud_cpp_${name}.a (or .so)
Create backwards compatibility physical libraries.

T + 6 months

Change the CMake files to only export the backwards compatibility targets if a macro is set (e.g. GOOGLE_CLOUD_CPP_DEFINE_LEGACY_TARGET_NAMES)

T + 12 months

Remove the GOOGLE_CLOUD_CPP_DEFINE_LEGACY_TARGET_NAMES macro.
Change the Bazel files to mark the targets in google/cloud/*/BUILD as private
This decommissions the legacy targets.

Open Items
What do we do about physical library names?
Renaming the libraries, even when we create backwards compatibility targets, will break users that use -l${lib} options directly.  There is not much we can do for these users:

There is no way to magically link an additional library, not when the libraries are static.
There is no way to generate a warning / deprecation message, not when the libraries are static.

Work Estimates
All estimates in "ideal hours", that is, uninterrupted work.

[x] Create a test to verify the existing CMake targets remain usable (3h)
[x] Create a test to verify the existing Bazel targets remain usable (3h)
[x] Create a test to verify the existing CMake targets in external/googleapis remain usable (3h)
[x] Mark all the existing Bazel targets as deprecated (1h)
[x] Create new Bazel targets for ${library}_client_mocks named //:${library}_mocks (3h)
[x] Rename the googleapis CMake exported targets to use google-cloud-cpp:: prefix (5h)
[x] Rename the bigtable CMake exported targets to use google-cloud-cpp:: prefix (5h)
[x] Rename the pubsub CMake exported targets to use google-cloud-cpp:: prefix (5h)
[x] Rename the spanner CMake exported targets to use google-cloud-cpp:: prefix (5h)
[x] Rename the storage CMake exported targets to use google-cloud-cpp:: prefix (5h)
[x] Rename the iam CMake exported targets to use google-cloud-cpp:: prefix (5h)
[x] Rename the logging CMake exported targets to use google-cloud-cpp:: prefix (5h)
[x] Create new release (2h)
[x] Push new release to vcpkg (1h)
[x] Use new targets in quickstart programs (1h)

Good idea. I like it. Thanks. Cleaning up and simplifying the build targets is a good idea, especially before we add NN new libraries from the generator. A few other thoughts to consider:

https://github.com/googleapis/google-cloud-cpp/issues/3701 suggests cleaning up bazel targets by deprecating //google/cloud:foo_client in favor of //:foo_client. We may want to lump the build target deprecations/simplifications together. That is, only if we agree that we want to proceed with #3701 at all.
While we're cleaning up build target names, we may want to consider bazel as well so that we end up with names that are mostly consistent even between build systems. I.e., whatever pattern we choose for cmake, we may want to follow the same pattern w/ bazel target naming. (note: this is independent of whether or not we do #3701)
Regarding the spelling of the prefix, though a mouth full, it may be unsurprising for users if we pick the project name google-cloud-cpp:: as the cmake prefix.
To shorten the names of the library slightly, we could consider omitting "_client". For example, google-cloud-cpp::lib${prefix}_{storage,spanner}.*

Good idea. I like it. Thanks. Cleaning up and simplifying the build targets is a good idea, especially before we add NN new libraries from the generator. A few other thoughts to consider:

https://github.com/googleapis/google-cloud-cpp/issues/3701 suggests cleaning up bazel targets by deprecating //google/cloud:foo_client in favor of //:foo_client. We may want to lump the build target deprecations/simplifications together. That is, only if we agree that we want to proceed with #3701 at all.
While we're cleaning up build target names, we may want to consider bazel as well so that we end up with names that are mostly consistent even between build systems. I.e., whatever pattern we choose for cmake, we may want to follow the same pattern w/ bazel target naming. (note: this is independent of whether or not we do #3701)
Regarding the spelling of the prefix, though a mouth full, it may be unsurprising for users if we pick the project name google-cloud-cpp:: as the cmake prefix.
To shorten the names of the library slightly, we could consider omitting "_client". For example, google-cloud-cpp::lib${prefix}_{storage,spanner}.*

Collecting some notes, I will update the description later:

We also have bigtable::client and storage::client alialies.

The bigtable protos appear as bigtable_protos in some places, and googleapis-c++::bigtable_protos.

use google-cloud-cpp:: instead of googleapis-c++::

We need to think about the changes to pkg-config

need to rename the files for find_package(storage_client) vs. find_package(google_cloud_cpp_common)

we need to offer backwards compatibility .pc and .cmake files

if we define aliases in the ${foo}-config.cmake files we need to remember to protect them with if(NOT TARGET ...) otherwise the user cannot do find_package(${foo}) multiple times.

We need to decide if we would rename any lib${bar}.a to libgoogle_cloud_cpp_${bar}.a. For these we could:

Create a file with a undefined symbol called lib${bar}_is_deprecated_use_libgoogle_cloud_cpp_${bar}_instead
That way users saying -l${bar} explicitly (not through pkg-config or CMake) would get a link error.

We need to add tests to verify these backwards compatibility files are properly installed.  A series of quickstart-like programs in the ci/kokoro/install builds might work.

We should probably start with these tests.

Collecting some notes, I will update the description later:

We also have bigtable::client and storage::client alialies.

The bigtable protos appear as bigtable_protos in some places, and googleapis-c++::bigtable_protos.

use google-cloud-cpp:: instead of googleapis-c++::

We need to think about the changes to pkg-config

need to rename the files for find_package(storage_client) vs. find_package(google_cloud_cpp_common)

we need to offer backwards compatibility .pc and .cmake files

if we define aliases in the ${foo}-config.cmake files we need to remember to protect them with if(NOT TARGET ...) otherwise the user cannot do find_package(${foo}) multiple times.

We need to decide if we would rename any lib${bar}.a to libgoogle_cloud_cpp_${bar}.a. For these we could:

Create a file with a undefined symbol called lib${bar}_is_deprecated_use_libgoogle_cloud_cpp_${bar}_instead
That way users saying -l${bar} explicitly (not through pkg-config or CMake) would get a link error.

We need to add tests to verify these backwards compatibility files are properly installed.  A series of quickstart-like programs in the ci/kokoro/install builds might work.

We should probably start with these tests.

@devjgm I have updated the description with the work I think we should do.  It includes renaming the physical libraries (which I think is necessary), and creating backwards compatibility libraries (which I am less certain of).

@devjgm I have updated the description with the work I think we should do.  It includes renaming the physical libraries (which I think is necessary), and creating backwards compatibility libraries (which I am less certain of).

More notes:

We need to decide if the Bazel targets stay @...google_cloud_cpp//:${library}_client or become @...google_cloud_cpp//:${library}
Maintaining backwards compatibility physical libraries for the googleapis-c++::${foo}_protos (in external/googleapis) is a pain.

More notes:

We need to decide if the Bazel targets stay @...google_cloud_cpp//:${library}_client or become @...google_cloud_cpp//:${library}
Maintaining backwards compatibility physical libraries for the googleapis-c++::${foo}_protos (in external/googleapis) is a pain.

FYI the packaging doc does not work as is on current master as you now have renamed the packages to google_cloud_cpp_bigtable etc. instead of google_cloud_bigtable

@ecourreges-orange thank you again for pointing out the problems in the documentation, as you probably noticed we merged a PR to fix the problem earlier today.  Let us know if you have any other problems.

This is done as of #8413

authors: coryan devjgm ecourreges-orange

repo: googleapis/google-cloud-cpp


text:
test(pubsub): update pubsub options tests

Make the publisher options default test check only that a default is set (instead of checking the exact value of a default)
Also add a mini test to ensure we are outputting warnings if an option not in the expected OptionList is supplied to the PublisherOptions constructor.

This change is

@coryan PTAL


text:
Request sample DialogFlow mega agent.

Yo,
Is there any possibility of adding an example using mega agents? I didn't find anything related to sub agents in QueryParams.
Thanks,
Cheers.

Assigning to Amanda who owns the samples repo, but only so that she can route the request to the Dialogflow team.

Closing in favor of GoogleCloudPlatform/dotnet-docs-samples/issues/1129

authors: amanda-tarafa jskeet juliolitwin

repo: googleapis/google-cloud-dotnet


text:
Synthesis failed for Google.Identity.AccessContextManager.Type

Hello! Autosynth couldn't regenerate Google.Identity.AccessContextManager.Type. :broken_heart:
Please investigate and fix this issue within 5 business days.  While it remains broken,
this library cannot be updated with changes to the Google.Identity.AccessContextManager.Type API, and the library grows
stale.
See https://github.com/googleapis/synthtool/blob/master/autosynth/TroubleShooting.md
for trouble shooting tips.
Here's the output from running synth.py:
2021-03-19 22:47:02,961 autosynth [INFO] > logs will be written to: /tmpfs/src/logs/Google.Identity.AccessContextManager.Type
2021-03-19 22:47:03,470 autosynth [DEBUG] > Running: git config --global core.excludesfile /home/kbuilder/.autosynth-gitignore
2021-03-19 22:47:03,473 autosynth [DEBUG] > Running: git config user.name yoshi-automation
2021-03-19 22:47:03,477 autosynth [DEBUG] > Running: git config user.email yoshi-automation@google.com
2021-03-19 22:47:03,481 autosynth [DEBUG] > Running: git config push.default simple
2021-03-19 22:47:03,484 autosynth [DEBUG] > Running: git branch -f autosynth-Google.Identity.AccessContextManager.Type
2021-03-19 22:47:03,488 autosynth [DEBUG] > Running: git checkout autosynth-Google.Identity.AccessContextManager.Type
Switched to branch 'autosynth-Google.Identity.AccessContextManager.Type'
2021-03-19 22:47:03,694 autosynth [DEBUG] > Running: git rev-parse --show-toplevel
2021-03-19 22:47:03,701 autosynth [DEBUG] > Running: git log -1 --pretty=%H
2021-03-19 22:47:03,705 autosynth [DEBUG] > Running: git remote get-url origin
2021-03-19 22:47:03,739 synthtool [ERROR] > Failed executing git checkout master:

error: Your local changes to the following files would be overwritten by checkout:
	google/cloud/dialogflow/v2/answer_record.proto
	google/cloud/dialogflow/v2/conversation.proto
	google/cloud/dialogflow/v2/conversation_profile.proto
	google/cloud/dialogflow/v2/participant.proto
Please commit your changes or stash them before you switch branches.
Aborting

Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 354, in <module>
    main()
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 189, in main
    return _inner_main(temp_dir)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 312, in _inner_main
    git_source.enumerate_versions(sources, pathlib.Path(temp_dir))
  File "/tmpfs/src/github/synthtool/autosynth/git_source.py", line 202, in enumerate_versions
    source_versions = enumerate_versions_for_source(git_source, temp_dir)
  File "/tmpfs/src/github/synthtool/autosynth/git_source.py", line 166, in enumerate_versions_for_source
    local_repo_dir = str(synthtool_git.clone(remote))
  File "/tmpfs/src/github/synthtool/synthtool/sources/git.py", line 85, in clone
    shell.run(["git", "checkout", "master"], cwd=str(dest), check=True)
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 39, in run
    raise exc
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'checkout', 'master']' returned non-zero exit status 1.

Google internal developers can see the full log here.

Autosynth passed, closing! :green_heart:


text:
StorageClient.UploadObject is difficult to use effectively

Describe the solution you'd like
The documentation claims that StorageClient.UploadObject[Async] is useful for situations where "You want to upload data but don't know the final size at the start of the upload, such as when generating the upload data from a process, or when compressing an object on-the-fly."  But the API it provides is not suited to either scenario.  Its working is the polar opposite of the standard .NET idiom for generating and storing data: the API gives you a writeable Stream, and you write the data to it.
Instead, StorageClient.UploadObject[Async] asks you for a readable Stream, and then uploads the data from it.  The example given in the linked doc page even involves using FileStream to upload an existing file from the file system, which is the precise opposite of generating data in-process where you don't know the final size at the start of the upload.  The scenario of "compressing an object on-the-fly" that the documentation claims to support does not work at all here; compressors such as GZipStream or BrotliStream require a writeable stream for their output.
While this API can certainly be useful for pre-existing data, it is clearly not fit for its advertised purpose.  A better API would look like this:
        public (Stream uploader, Task<Object> receipt) BeginUploadObject(
            string bucket,
            string objectName,
            string contentType,
            UploadObjectOptions options = null)

(plus a comparable Async variant)

This would return a writeable Stream to write data to.  The API will know that the data production is complete when Close or Dispose is called on the stream.  Generation of the Object metatata that serves as a receipt for your object storage would be the responsibility of an async Task, even for the synchronous API.
Describe alternatives you've considered
Right now, the only way to upload generated data without buffering it either in memory or in a temp file is to come up with some sort of hack to adapt a writeable Stream to a readable Stream.  System.IO.Pipelines.Pipe seems to work, but that imposes additional overhead for moving the uploading code onto a thread pool thread.  It would be nice to instead have an API that's actually designed to fit the advertised workflow.

@bajajneha27 - Can you look into this?

First off, @masonwheeler, thanks for raising this - and I do completely agree that the library API surface is clunky here. Likewise it would be nice if DownloadObject could provide a stream to read from. I'm not sure that we'd want to expose a ValueTuple-based return type, but other than that your proposed API looks about right to me.
Unfortunately, implementing that API presents significant challenges. The existing implementation is based on ResumableUpload from the Google.Apis package - which only supports the current mode, and is a hairy piece of code to work with. (It's worth understanding that Google.Apis is effectively support for HTTP/1.1+JSON API calls. It's in maintenance mode - we're actively maintaining it, but not putting effort into adding features unless we absolutely need to. The newer generation of gRPC-based libraries is where most of our effort goes at the moment.)
There are three options we could use to move forward:

Continue to use ResumableUpload but use it with the kind of buffer you describe at the end. (With the overhead you describe as well. Note that there's likely to be some memory overhead, as in order to provide retry we'd need to keep the data locally until we've made a successful request.)
Add the functionality to ResumableUpload: this would be really complex, unfortunately.
Create an entirely separate path that doesn't use ResumableUpload at all - but needs to implement a lot of the same functionality

None of these is appealing, as you can imagine.
We'll need to discuss internally how important it is to fix this, estimate timelines, and see how it fits in with any other future plans. I don't want to make any promises that we'll ever address this - but I definitely acknowledge that the issue is a real one.

@jskeet

I'm not sure that we'd want to expose a ValueTuple-based return type, but other than that your proposed API looks about right to me.

Fair enough.  The other alternative is to use an out parameter, but that gets awkward when you also have params with default values.

It's worth understanding that Google.Apis is effectively support for HTTP/1.1+JSON API calls. ... The newer generation of gRPC-based libraries is where most of our effort goes at the moment.

Huh.  I'm actually a bit surprised to hear that, well over a decade after they were introduced, Google hasn't provided a WebSocket-based upload API.  You'd think that would be the obvious implementation for this particular use case.

None of these is appealing, as you can imagine.

True.  Well, of those three, I have to ask, does the buffer you describe as being necessary for making the resumable upload resumable already exist, seeing as how it's necessary?  (Or does your implementation depend on Stream.Seek and explode when it's not available?  Because I think that's the only other alternative, and that is not documented as being the case.)  If the necessary buffer is already there, option 1 easily looks like the lesser of three evils.

On a quick investigation, it looks like we always buffer - I had thought we might only buffer if Stream.CanSeek returns false, but it looks like that's not currently the case. (We definitely don't explode when seeking isn't available.)
So option 1 would be no worse than the current code in terms of overhead - it just wouldn't reduce the overhead either.
(In terms of the API, there are various options here; we'll think about that after deciding whether or not to actually go ahead at all.)

We definitely don't explode when seeking isn't available.

Well congrats on that; that puts you a step ahead of your AWS S3 counterpart!

I'm looking at this now, with a tentative design that's basically a stream to perform the buffering - so you might have something like:
var mediaStream = new MediaStream(/* buffer size maybe? */);
// Important - we *don't* await this task yet...
var uploadTask = client.UploadObjectAsync("my-bucket", "my-object", "text/plain", mediaStream);
// We'd probably *only* support async code for this. It makes things simpler.
await mediaStream.WriteAsync(...);
mediaStream.Close(); // Or Dispose, or a different method entirely...

// This would only finish when the buffered data had all been pulled by the uploader
await uploadTask;

We might be able to use the same class (or at least bits of it) for a cleaner download as well (where you read from the download stream). Or we may be able to use an entirely different design for that. (That gets tricky in terms of hash checking.)
It's still a bit clunky - more so now that I'm focused on the user code rather than the implementation. Hmm.

Honestly that setup looks like a footgun, if a minor one.  "await SomeApiAsync()" is so ingrained into .NET developers' heads by this point that I can see a lot of people simply doing that automatically and not understanding why it didn't work, which seems at odds with the "pit of success" philosophy underlying .NET design.
What happens if you do await the call to UploadObjectAsync?  I'm guessing it will simply never complete?

@masonwheeler: Yup, that's my big concern. And yes, it would just never complete because it would be waiting for the stream to be completed. I'm starting to think about different options (which may basically use that underneath)...

@jskeet Aside from the ValueTuple, which could probably be worked around in a few different ways, is there any reason why the API I suggested in the first post wouldn't be viable?

@masonwheeler: I think I'd probably want to wrap it into a separate type. There's still the concern that someone will see a task and just await it though. I don't want to leap to one suggested API rather than trying to think of multiple options - and ones that will work for other media upload/download in Google.Apis as well.

Apologies for the silence on this for a couple of months - and apologies that you're not going to like the result, which is that I'm going to close this issue with a new PR that adds this to our backlog file. I've looked into it, and I just don't think there's anything nice we can do in the current library. If we end up with a new Storage library based on gRPC, we can look at it from scratch at that point, as we'll have a clean slate in terms of API surface - we won't need to worry about how the existing REST-based support library works, and we won't need to make changes to the existing API of Google.Cloud.Storage.V1. You have my word that we'll at least take another close look - and hopefully with that added freedom, we'll be able to come up with something appropriate. I think it's unlikely that we'll ever end up back-porting to Google.Cloud.Storage.V1 though.
I know this isn't the news you want to hear, but I'm afraid we have limited resources in the team, and implementing a robust and "clean" API (avoiding traps) here would just be too expensive in terms of time.

Wow.  "We have limited resources" is kind of the last thing you expect to ever hear from Google... 😮

I know this isn't the news you want to hear, but I'm afraid we have limited resources in the team, and implementing a robust and "clean" API (avoiding traps) here would just be too expensive in terms of time.

A possible way to avoid traps is to add another method that accepts System.IO.Pipelines.PipeReader and continuously uploads chunks from pipe until the pipe is completed by another thread with .Complete() method.

authors: groone jskeet masonwheeler shivgautam


text:
Release Google.Cloud.Batch.V1Alpha version 1.0.0-alpha21

Changes in this release:
Bug fixes

BREAKING CHANGE Remove deprecated field enableOslogin (commit 1531c7e)

Documentation improvements

Polish the field descriptions for enableImageStreaming and CloudLoggingOptions (commit 5570490)

Created release for Google.Cloud.Batch.V1Alpha-1.0.0-alpha21

authors: amanda-tarafa yoshi-automation


text:
Refresh Compute Java client.

Regenerated with https://github.com/googleapis/discovery-artifact-manager/pull/100. That must be merged in first.

Should I not have accepted #4201? If not, we can wait for autosynth to run tomorrow morning.

@chingor13 Oh I didn't see that PR. No rush. The updated GAPIC config that I just checked might have a couple more methods added.

repo: googleapis/google-cloud-java


text:
Regenerate dlp client

This PR was generated using Autosynth. :rainbow:
Here's the log from Synthtool:
synthtool > Executing /tmpfs/src/git/autosynth/working_repo/google-cloud-clients/google-cloud-dlp/synth.py.
synthtool > Ensuring dependencies.
synthtool > Pulling artman image.
latest: Pulling from googleapis/artman
Digest: sha256:80c39fa84e7203c8f355e01bdeef82155013cc39dcaa48fba7a6fe2c253623e3
Status: Image is up to date for googleapis/artman:latest
synthtool > Cloning googleapis.
synthtool > Running generator for google/privacy/dlp/artman_dlp_v2.yaml.
synthtool > Generated code into /home/kbuilder/.cache/synthtool/googleapis/artman-genfiles/java.
synthtool > Running java formatter on 10 files
synthtool > Running java formatter on 1 files
synthtool > Running java formatter on 285 files
synthtool > Cleaned up 0 temporary directories.
synthtool > Wrote metadata to synth.metadata.

Just formatting changes - closing.


text:
When possible, references to protobuf messages should be rendered as links to the generated class

For more context: https://github.com/GoogleCloudPlatform/google-cloud-php/pull/1113#discussion_r198255098

This still exists in LongRunning client, but I cannot trace how this was fixed. cc:@bshaffer

I don't understand the request here, and the context link is not helpful either. Can you explain more what the objective is here?

I think this is outdated. We now use @see tags, which are helpful for IDEs and also for our documentation at cloud.google.com/php/docs/reference. I do not think there's anything else that needs to be done, and we can close this.

authors: bshaffer dwsupplee vishwarajanand

repo: googleapis/google-cloud-php