text:
GraalVM native image and netty's Slf4JLoggerFactory from gax-grpc/native-image.properties

I'm trying to build a native image that uses gcloud java libraries. The first issue I encountered is
No instances of io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory
are allowed in the image heap as this class should be initialized at image runtime.

Fatal error: org.graalvm.compiler.debug.GraalError: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: No instances of io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory are allowed in the image heap as this class should be initialized at image runtime. Object has been initialized by the io.grpc.netty.shaded.io.netty.channel.ChannelOption class initializer with a trace:
 	at io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory.<init>(Slf4JLoggerFactory.java:42)
	at io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory.<clinit>(Slf4JLoggerFactory.java:33)
	at io.grpc.netty.shaded.io.netty.util.internal.logging.InternalLoggerFactory.useSlf4JLoggerFactory(InternalLoggerFactory.java:62)
	at io.grpc.netty.shaded.io.netty.util.internal.logging.InternalLoggerFactory.newDefaultFactory(InternalLoggerFactory.java:42)
	at io.grpc.netty.shaded.io.netty.util.internal.logging.InternalLoggerFactory.getDefaultFactory(InternalLoggerFactory.java:111)
	at io.grpc.netty.shaded.io.netty.util.internal.logging.InternalLoggerFactory.getInstance(InternalLoggerFactory.java:134)
	at io.grpc.netty.shaded.io.netty.util.internal.logging.InternalLoggerFactory.getInstance(InternalLoggerFactory.java:127)
	at io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:81)
	at io.grpc.netty.shaded.io.netty.util.ConstantPool.<init>(ConstantPool.java:34)
	at io.grpc.netty.shaded.io.netty.channel.ChannelOption$1.<init>(ChannelOption.java:36)
	at io.grpc.netty.shaded.io.netty.channel.ChannelOption.<clinit>(ChannelOption.java:36)
.  To fix the issue mark io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory for build-time initialization with --initialize-at-build-time=io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory or use the the information from the trace to find the culprit and --initialize-at-run-time=<culprit> to prevent its instantiation.

when I tried to apply the suggested fix I get another error that points to gax-grpc:
Error: Incompatible change of initialization policy for io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory:
trying to change RUN_TIME from 'META-INF/native-image/com.google.api/gax-grpc/native-image.properties' in 'file:///.../com.google.api/gax-grpc/2.18.1/a3f2cca007aaa26868b4750e56b0af7ae63f5e3c/gax-grpc-2.18.1.jar' with 'io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory'
to BUILD_TIME from command line with 'io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory'
com.oracle.svm.core.util.UserError$UserException:
Incompatible change of initialization policy for io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory: trying to change RUN_TIME from 'META-INF/native-image/com.google.api/gax-grpc/native-image.properties'
in 'file:///.../com.google.api/gax-grpc/2.18.1/a3f2cca007aaa26868b4750e56b0af7ae63f5e3c/gax-grpc-2.18.1.jar' with 'io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory' to BUILD_TIME from command line with 'io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory'
	at com.oracle.svm.core.util.UserError.abort(UserError.java:72)
	at com.oracle.svm.hosted.classinitialization.ClassInitializationConfiguration.insertRec(ClassInitializationConfiguration.java:102)

implementation platform("com.google.cloud:spring-cloud-gcp-dependencies:3.3.0")
implementation group: 'com.google.cloud', name: 'spring-cloud-gcp-starter-trace'
implementation "com.google.api:gax:2.18.1"
implementation "com.google.api:gax-grpc:2.18.1"

Hi @edudar, thanks for filing this issue! Are you experiencing this with only spring-cloud-gcp-starter-trace or with any other libraries?

@mpeddada1 it's not specific to tracing, that just happened to be the library that is adopted the widest in our codebase. Having these two lines in the build is enough for it to fail:
implementation "com.google.api:gax:2.18.1"
implementation "com.google.api:gax-grpc:2.18.1"

Here are some log excerpts (I use 22.2 because of M1 Mac):
[1/7] Initializing...                                                                       (5.3s @ 0.30GB)
 Version info: 'GraalVM 22.2.0-dev Java 17 CE'
 C compiler: cc (apple, arm64, 13.1.6)
 Garbage collector: Serial GC
 5 user-specific feature(s)
 - com.google.api.gax.grpc.nativeimage.GrpcNettyFeature
 - com.google.api.gax.grpc.nativeimage.ProtobufMessageFeature
 - com.google.api.gax.nativeimage.GoogleJsonClientFeature
 - com.google.api.gax.nativeimage.OpenCensusFeature
 - com.oracle.svm.thirdparty.gson.GsonFeature
...
13:44:14.426 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
13:44:14.426 [main] DEBUG io.grpc.netty.shaded.io.netty.util.ResourceLeakDetector - -Dio.grpc.netty.shaded.io.netty.leakDetection.level: simple
13:44:14.426 [main] DEBUG io.grpc.netty.shaded.io.netty.util.ResourceLeakDetector - -Dio.grpc.netty.shaded.io.netty.leakDetection.targetRecords: 4
13:44:14.511 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - -Dio.netty.noUnsafe: false
13:44:14.511 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - Java version: 17
13:44:14.512 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
13:44:14.512 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
13:44:14.512 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
13:44:14.512 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
13:44:14.512 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: unavailable, false, Reflective setAccessible(true) disabled
13:44:14.512 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): available
13:44:14.512 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): unavailable
13:44:14.512 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available
13:44:14.512 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - maxDirectMemory: 13362528256 bytes (maybe)
13:44:14.512 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /var/folders/n6/pp4krd7d41d623_tbh6dq8x00000gn/T (java.io.tmpdir)
13:44:14.513 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
13:44:14.513 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - Platform: MacOS
13:44:14.513 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - -Dio.netty.maxDirectMemory: -1 bytes
13:44:14.513 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - -Dio.netty.uninitializedArrayAllocationThreshold: 1024
13:44:14.513 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.CleanerJava9 - java.nio.ByteBuffer.cleaner(): available
13:44:14.513 [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
13:44:14.559 [main] DEBUG io.grpc.netty.shaded.io.netty.channel.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 20
...
13:44:18.152 [main] DEBUG io.grpc.netty.shaded.io.netty.buffer.AbstractByteBuf - -Dio.grpc.netty.shaded.io.netty.buffer.checkAccessible: true
13:44:18.152 [main] DEBUG io.grpc.netty.shaded.io.netty.buffer.AbstractByteBuf - -Dio.grpc.netty.shaded.io.netty.buffer.checkBounds: true
13:44:18.152 [main] DEBUG io.grpc.netty.shaded.io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.grpc.netty.shaded.io.netty.util.ResourceLeakDetector@5306300e
13:44:27.997 [ForkJoinPool-2-worker-4] DEBUG io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.workdir: /var/folders/n6/pp4krd7d41d623_tbh6dq8x00000gn/T (io.netty.tmpdir)
13:44:27.997 [ForkJoinPool-2-worker-4] DEBUG io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.deleteLibAfterLoading: true
13:44:27.997 [ForkJoinPool-2-worker-4] DEBUG io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.tryPatchShadedId: true
13:44:28.195 [ForkJoinPool-2-worker-8] DEBUG io.grpc.netty.shaded.io.netty.buffer.AdvancedLeakAwareByteBuf - -Dio.grpc.netty.shaded.io.netty.leakDetection.acquireAndReleaseOnly: false
...
Fatal error: org.graalvm.compiler.debug.GraalError: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: No instances of io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory are allowed in the image heap as this class should be initialized at image runtime. Object has been initialized by the io.grpc.netty.shaded.io.netty.buffer.AbstractByteBufAllocator class initializer with a trace:
 	at io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory.<init>(Slf4JLoggerFactory.java:42)
	at io.grpc.netty.shaded.io.netty.util.internal.logging.Slf4JLoggerFactory.<clinit>(Slf4JLoggerFactory.java:33)
	at io.grpc.netty.shaded.io.netty.util.internal.logging.InternalLoggerFactory.useSlf4JLoggerFactory(InternalLoggerFactory.java:62)

That is interesting. I created a small sample app with these two dependencies and I haven't been able to reproduce this issue on my end: https://github.com/mpeddada1/simple-netty-test. Do you have a sample that we can try out? It might help with finding the root cause of the issue.

@mpeddada1 might work because there's no slf4j dependency in your sample project. Check this one out: https://github.com/edudar/gax-java-native-slf4jlogger

Same issue here! Does somebody found a workaround? this prevents upgrading dependencies.

@mpeddada1 Would it possible to make a release that includes the fix?

@mpeddada1 looks good so far. I managed to run my test project with two additional initialization:
'--initialize-at-build-time=org.slf4j.impl.SimpleLogger'
'--initialize-at-build-time=org.slf4j.impl.StaticLoggerBinder'

But it compiles and runs!

authors: edudar lpellegr mpeddada1

repo: googleapis/gax-java


text:
Fix javadocs so that both gax and gax-grpc docs are included

Currently, only gax-grpc is showing up in the javadocs.

Javadocs have been aggregating for a while now.

source: gharchive/issue

license_type: permissive

license_source: bigquery


text:
GaxError presents BadStatus values

This PR relays the underlying BadStatus values code, details, and metadata without resorting to dynamic hackery using method_missing. If the underlying error object does not have values, or is not a BadStatus, nil is returned.
[refs GoogleCloudPlatform/google-cloud-ruby#1791]
[refs GoogleCloudPlatform/google-cloud-ruby#1825]

Codecov Report

Merging #104 into master will increase coverage by 0.08%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #104      +/-   ##
==========================================
+ Coverage   99.35%   99.44%   +0.08%
==========================================
  Files          18       19       +1
  Lines        1708     1794      +86
==========================================
+ Hits         1697     1784      +87
+ Misses         11       10       -1

Impacted Files
Coverage Δ

lib/google/gax/errors.rb
100% <100%> (ø)
:arrow_up:

spec/google/gax/error_spec.rb
100% <100%> (ø)

lib/google/gax/grpc.rb
100% <0%> (+2.17%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e9e15f...9e907ee. Read the comment docs.

Build is fixed. Thank you for your patience.

source: gharchive/pull-request

authors: blowmage codecov-io

repo: googleapis/gax-ruby


text:
File uploaded without extension, and didn't load directly into the analytics

I use this code to upload a report file in analytics:
$client = new \Google_Client();
$client->setApplicationName("Hello Analytics Reporting");
$client->setAuthConfig($KEY_FILE_LOCATION);
$client->setScopes(['https://www.googleapis.com/auth/analytics']);
$this->analytics = new \Google_Service_Analytics($client);

$this->analytics->management_uploads->uploadData(
                $this->config['accountId'],
                $this->config['webPropertyId'],
                $this->config['customDataSourceId'],
                array('data' => file_get_contents($reportFile),
                    'mimeType' => 'application/octet-stream',
                    'uploadType' => 'media'));

So, file successfully uploaded here: http://prntscr.com/xzyej2 http://prntscr.com/xzyf62 http://prntscr.com/xzyfey http://prntscr.com/xzyfn5 , but it has different name, like HJ7iaazwS_ud_JSMKy8zwQ without extension.
It is completely identical report-file, i have got on site, except name and extension.
The error is that the data from the file is not loaded directly into the analytics (http://prntscr.com/xzyhjc)
What am I doing wrong?

You seem to be looking for guidance on how to use a specific API. We (the maintainers of this repository) only provide support for the code in this repository. Consider asking your question on Stack Overflow using some of the following tags:

google-analytics-api
google-api
google-cloud-platform
google-cloud-sdk

If you have a paid support plan with Google Cloud, you can open a support ticket.

repo: googleapis/google-api-php-client


text:
Replace bigtable::internal::conjunction with absl::conjunction.

The bigtable::internal::conjunction<> (currently bigtable::detail::conjunction<>, will change with #36) template class is a simple implementation of std::conjunction<> introduced with C++17.
On environments where std::conjunction<> is available we should replace the class with an alias.  That will ensure our code is not depending on some particular hack in our implementation.

As an alternative, apparently Abseil provides absl::conjunction

FWIW, we have a dep on Abseil now, so we could use absl::conjunction

Looks like @devjgm fixed this in #4826

authors: coryan devjgm

repo: googleapis/google-cloud-cpp

license_source: github-api


text:
BigQuery returns NULLABLE mode for a field that has a REQUIRED mode (field created as a NOT NULL from the beginning)

Environment details
BigQuery Client Library for Java: google-cloud-bigquery version 2.37.2 and below
I'm using Java 11.
Steps to reproduce

Create a table in BigQuery console (or in any other way) with a NOT NULL field, i.e. CREATE TABLE sample.TEST_TABLE ( C1 INT64 NOT NULL, C2 BIGNUMERIC );
Use BigQuery Client Library for Java to execute select statement for that table. Received schema should report that C1 column has mode REQUIRED, while it reports the mode as NULLABLE.

Code example
// create a com.google.cloud.bigquery.BigQuery object, i.e.:
// BigQuery bigQuery = BigQueryOptions.newBuilder()......

String query = "SELECT * FROM sample.TEST_TABLE";
Job job = bigQuery.create(JobInfo.of(QueryJobConfiguration.newBuilder(query).build()));
TableResult result = job.getQueryResults();
Schema schema = result.getSchema();

// Print the schema, or print the fields:
List<Field> fields = schema.getFields();
for (Field field : fields) {
    System.out.println("Field Mode: " + field.getMode());
    // C1 column which was created as NOT NULL and has REQUIRED mode in BigQuery console
    // is being returned here as mode NULLABLE
}

// OR
// create a com.google.cloud.bigquery.BigQuery object, i.e.:
// BigQuery bigQuery = BigQueryOptions.newBuilder()......

String query = "SELECT * FROM sample.TEST_TABLE";
QueryJobConfiguration.Builder configuration = QueryJobConfiguration.newBuilder(query).setDryRun(true);
Schema schema = ((QueryStatistics) bigQuery.create(JobInfo.of(configuration.build())).getStatistics()).getSchema();

System.out.println(schema);
// See at C1 column mode: Should be REQUIRED but is NULLABLE.

Prints out 2 nullable fields.

When a query is sent to BigQuery to be run, an anonymous table is created to hold the query results. This anonymous table is separate from the existing table created in the console (or any other method). When the anonymous table is created, all fields are set to be NULLABLE by default as it is not always possible to guarantee that the values in the anonymous table would not be NULL. This anonymous table is created and used specifically to hold the results of a query, and does not affect or modify the existing table used to run the query.
In the above code examples, the schema is actually read from the anonymous table rather than the permanent existing table created in the console. We can see this as it is being constructed and copied from the job:
TableResult result = job.getQueryResults();

Because of this, the schema will always have all field modes set to NULLABLE, regardless of the schema of the original table. This is intended behavior.
In order to get the schema of the original table, you can use other methods that specify the dataset and table IDs. For example:
BigQuery bq = BigQueryOptions.newBuilder().setProjectId(<your-project-id>).build().getService();
Table table = bq.getTable(<your-dataset-id>, <your-table-id>);
Schema schema = Schema.fromPb(table.toPb().getSchema());

This schema refers to the original table and will be consistent with the schema specified in the console. Normally, the fact that the anonymous table is created in the background is not very relevant to the user, but it is important to note that the query results are stored in a table separate from the table used to actually run the query.

repo: googleapis/java-bigquery


text:
[CHANGE ME] Re-generated to pick up changes from synthtool.

This PR was generated using Autosynth. :rainbow:
Synth log will be available here:
https://source.cloud.google.com/results/invocations/a79f8a17-dbb5-42d4-99e9-73d2b5ffda4d/targets

[ ] To automatically regenerate this PR, check this box.

Source-Link: https://github.com/googleapis/synthtool/commit/9602086c6c5b05db77950c7f7495a2a3868f3537
Source-Link: https://github.com/googleapis/synthtool/commit/c3caf0704f25a0c365f1c315e804a30b87c62a75

@chingor13 -- there are no samples in this repo. should we exclude this yaml file?

yeah we should exclude it here. the job probably should be more resilient, but let's exclude it

repo: googleapis/java-cloud-bom


text:
[CHANGE ME] Re-generated to pick up changes from self.

This PR was generated using Autosynth. :rainbow:
Synth log will be available here:
https://source.cloud.google.com/results/invocations/3f35c52e-e8a4-495e-8072-959688955792/targets

[ ] To automatically regenerate this PR, check this box.

@simonz130 The CODEOWNERS file is managed by templates. The intention of that line is to ensure the java-samples readability folks approve sample changes. If we want to include api-logging as an owner of those files, we can add a template exclude in synth.py

repo: googleapis/java-logging


text:
io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Invalid data in message: Message failed schema validation.

I have this Kafka topic named source which contains AVRO messages already derived from the schema registry on http://localhost:9080. Producing the messages was as (see code below) to which the messages can be viewed by the Kafka UI (through the schema registry), hence (as I believe) the schema registry is also working as expected.
ProducerRecord<String, AvroCustomMessage> record = new ProducerRecord<>("source", "test", customMessage);
kafkaTemplate.send(record);

However, when I am setting up a Sink PubSub Connector from the source topic to cps.topic.test using the connector below. Am getting these stacktrace
Connector config
{
  "name": "CONNECTOR_NAME"
  "connector.class": "com.google.pubsub.kafka.sink.CloudPubSubSinkConnector",
  "headers.publish": true,
  "metadata.publish": true,
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "key.converter.schemas.enable": false,
  "key.converter.schemas.registry.url": "http://localhost:9081",
  "key.converter.enhanced.avro.schema.support": true,
  "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
  "value.converter.schemas.enable": false,
  "value.converter.schemas.registry.url":" http://localhost:9081",
  "value.converter.enhanced.avro.schema.support": true,
  "tasks.max": "1",
  "topics": "source",
  "cps.topic": "cps.topic.test",
  "cps.project": "cps.project.test",
  "cps.endpoint": "pubsub.googleapis.com:443",
  "gcp.credentials.file.path": "path/to/file.json"
}

Error
2023-06-15T14:19:03.967837839Z [2023-06-15 14:19:03,967] DEBUG [CONNECTOR_NAME|task-0] Flushing... (com.google.pubsub.kafka.sink.CloudPubSubSinkTask:329)
2023-06-15T14:19:03.967846553Z [2023-06-15 14:19:03,967] TRACE [CONNECTOR_NAME|task-0] Received flush for partition source-0 (com.google.pubsub.kafka.sink.CloudPubSubSinkTask:333)
2023-06-15T14:19:03.971307523Z [2023-06-15 14:19:03,968] ERROR [CONNECTOR_NAME|task-0] WorkerSinkTask{id=CONNECTOR_NAME-0} Offset commit failed, rewinding to last committed offsets (org.apache.kafka.connect.runtime.WorkerSinkTask:409)
2023-06-15T14:19:03.971340639Z java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Invalid data in message: Message failed schema validation.
2023-06-15T14:19:03.971372375Z 	at com.google.pubsub.kafka.sink.CloudPubSubSinkTask.flush(CloudPubSubSinkTask.java:347)
2023-06-15T14:19:03.971386314Z 	at org.apache.kafka.connect.sink.SinkTask.preCommit(SinkTask.java:125)
2023-06-15T14:19:03.971393094Z 	at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:404)
2023-06-15T14:19:03.971398732Z 	at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:374)
2023-06-15T14:19:03.971404374Z 	at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:218)
2023-06-15T14:19:03.971409965Z 	at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:203)
2023-06-15T14:19:03.971414840Z 	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:188)
2023-06-15T14:19:03.971419642Z 	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)
2023-06-15T14:19:03.971424466Z 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2023-06-15T14:19:03.971429693Z 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-06-15T14:19:03.971434557Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2023-06-15T14:19:03.971439081Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2023-06-15T14:19:03.971443774Z 	at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-15T14:19:03.971448940Z Caused by: java.util.concurrent.ExecutionException: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Invalid data in message: Message failed schema validation.
2023-06-15T14:19:03.971454003Z 	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
2023-06-15T14:19:03.971477053Z 	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)
2023-06-15T14:19:03.971482768Z 	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:113)
2023-06-15T14:19:03.971490433Z 	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:66)
2023-06-15T14:19:03.971495828Z 	at com.google.pubsub.kafka.sink.CloudPubSubSinkTask.flush(CloudPubSubSinkTask.java:345)
2023-06-15T14:19:03.971500928Z 	... 12 more
2023-06-15T14:19:03.971505860Z Caused by: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Invalid data in message: Message failed schema validation.
2023-06-15T14:19:03.971511459Z 	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
2023-06-15T14:19:03.971516728Z 	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
2023-06-15T14:19:03.971557435Z 	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
2023-06-15T14:19:03.971571292Z 	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
2023-06-15T14:19:03.971576787Z 	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
2023-06-15T14:19:03.971581325Z 	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
2023-06-15T14:19:03.971648068Z 	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
2023-06-15T14:19:03.971654268Z 	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
2023-06-15T14:19:03.971659263Z 	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
2023-06-15T14:19:03.971663692Z 	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
2023-06-15T14:19:03.971668328Z 	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:574)
2023-06-15T14:19:03.971687253Z 	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:544)
2023-06-15T14:19:03.971693772Z 	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
2023-06-15T14:19:03.971696819Z 	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
2023-06-15T14:19:03.971700104Z 	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
2023-06-15T14:19:03.971704966Z 	at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:541)
2023-06-15T14:19:03.971710298Z 	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489)
2023-06-15T14:19:03.971715273Z 	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453)
2023-06-15T14:19:03.971732361Z 	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486)
2023-06-15T14:19:03.971740825Z 	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:576)
2023-06-15T14:19:03.971744216Z 	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
2023-06-15T14:19:03.971747150Z 	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:757)
2023-06-15T14:19:03.971750185Z 	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:736)
2023-06-15T14:19:03.971753106Z 	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2023-06-15T14:19:03.971756012Z 	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
2023-06-15T14:19:03.971758846Z 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2023-06-15T14:19:03.971761676Z 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-06-15T14:19:03.971764526Z 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
2023-06-15T14:19:03.971767486Z 	... 3 more
2023-06-15T14:19:03.971770336Z Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Invalid data in message: Message failed schema validation.
2023-06-15T14:19:03.971773264Z 	at io.grpc.Status.asRuntimeException(Status.java:539)
2023-06-15T14:19:03.971776331Z 	... 20 more
2023-06-15T14:19:03.971780978Z

What seems to be the issue why am getting the error message?

We will need more details to be able to look into this for you - specifically we will need to look at the Pub/Sub topic schema and  the sample message you are trying to send. If your schema/data is confidential, you can try to reproduce the error  with non-confidential data.

I run into similar issue.  I have a avro schema for my kafka topic. I created the same schema for my pubsub topic (copied the schema from my kafka schema registry and used it as gcp pubsub schema for the pubsub topic). I tested the same message that is in the kafka topic directly by copying the same message and publish the message in gcp pubsub topic by using the gcp web console and the message is published.
However, sink connector (version 1.2.0) is failing...
Connector config:
{ 	"connector.class": "com.google.pubsub.kafka.sink.CloudPubSubSinkConnector", 	"errors.log.include.messages": "true", 	"tasks.max": "1", 	"topics": "klarna.reports", 	"cps.project": "mycps-project-id", 	"gcp.credentials.file.path": "******", 	"key.converter.schemas.enable": "false", 	"value.converter.schema.registry.url": "http://127.0.0.1:8081", 	"name": "pubsub-sink-klarna-reports", 	"value.converter": "io.confluent.connect.avro.AvroConverter", 	"cps.topic": "klarna.reports", 	"errors.log.enable": "true", 	"key.converter": "org.apache.kafka.connect.storage.StringConverter" }
Stacktrace:
{"timestamp":"2024-04-03 14:38:17,887","level":"ERROR","appName":"kafka-connect","thread":"task-thread-pubsub-sink-prada-application-status-fixed-interest-0","class":"org.apache.kafka.connect.runtime.WorkerSinkTask","message":"WorkerSinkTask{id=pubsub-sink-prada-application-status-fixed-interest-0} Commit of offsets threw an unexpected exception for sequence number 8: null java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Invalid data in message: Message failed schema validation. 	at com.google.pubsub.kafka.sink.CloudPubSubSinkTask.flush(CloudPubSubSinkTask.java:352) 	at org.apache.kafka.connect.sink.SinkTask.preCommit(SinkTask.java:125) 	at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:378) 	at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:208) 	at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192) 	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177) 	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227) 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 	at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.util.concurrent.ExecutionException: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Invalid data in message: Message failed schema validation. 	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588) 	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) 	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:113) 	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:66) 	at com.google.pubsub.kafka.sink.CloudPubSubSinkTask.flush(CloudPubSubSinkTask.java:350) 	... 11 more Caused by: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Invalid data in message: Message failed schema validation. 	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92) 	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98) 	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66) 	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) 	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67) 	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132) 	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) 	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270) 	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038) 	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808) 	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:574) 	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:544) 	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) 	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) 	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) 	at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:541) 	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:576) 	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) 	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:757) 	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:736) 	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) 	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) 	... 3 more Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Invalid data in message: Message failed schema validation. 	at io.grpc.Status.asRuntimeException(Status.java:539) 	... 17 more "}
I have tested the connector and it is sinking the data into pubsub topic when pubsub topic does not use any schema. However, the kafka message payload is not stored in the Message Body(pubsub topic), rather all the fields in the message payload is exploded as attribute. .

I have tested for a simple avro schema that I created for both my new test kafka topic and pubsub topic that is as follows:
{
"type": "record",
"name": "TesPubsubSchemaRecord",
"namespace": "se.kf.medmera.common",
"fields": [
{
"name": "id",
"type": "int"
},
{
"name": "name",
"type": "string"
},
{
"name": "phone",
"type": "string",
"default": "123"
},
{
"name": "lastname",
"type": "string",
"default": null
}
]
}
But getting the same Schema validation error

authors: dpbadiola samarthsingal xsajkha


text:
Google AutoML Error - Error: 4 DEADLINE_EXCEEDED: Deadline exceeded

Environment details

OS: Ubuntu 18.04
Node.js version: v11.15.0
npm version: 6.13.2
@google-cloud/automl version: 1.10.0

Steps to reproduce
I am using the nodejs library to import data to the dataset using the following code.
require("dotenv").config();
const automl = require("@google-cloud/automl");
const client = new automl.v1beta1.AutoMlClient();

(async () => {
  const projectId = `xxx`;
  const computeRegion = `us-central1`;
  const datasetId = `xxx`;
  const path = "gs://example_bucket_ocr_idr2/happiness.csv";

  // Get the full path of the dataset.
  const datasetFullId = client.datasetPath(projectId, computeRegion, datasetId);

  let inputConfig = {};
  if (path.startsWith(`bq`)) {
    // Get Bigquery URI.
    inputConfig = {
      bigquerySource: {
        inputUri: path
      }
    };
  } else {
    // Get the multiple Google Cloud Storage URIs.
    const inputUris = path.split(`,`);
    inputConfig = {
      gcsSource: {
        inputUris: inputUris
      }
    };
  }

  // Import the dataset from the input URI.
  client
    .importData({ name: datasetFullId, inputConfig: inputConfig })
    .then(responses => {
      const operation = responses[0];
      console.log(`Processing import...`);
      return operation.promise();
    })
    .then(responses => {
      // The final result of the operation.
      const operationDetails = responses[2];

      // Get the data import details.
      console.log("Data import details:");
      console.log(`\tOperation details:`);
      console.log(`\t\tName: ${operationDetails.name}`);
      console.log(`\t\tDone: ${operationDetails.done}`);
    })
    .catch(err => {
      console.error(err);
    });
})();

But every time I run it, I get DEADLINE_EXCEEDED error. I have searched for a solution but didnt find any till now.
{ Error: 4 DEADLINE_EXCEEDED: Deadline exceeded
    at Object.callErrorFromStatus (/var/www/html/test/google-ml/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Http2CallStream.call.on (/var/www/html/test/google-ml/node_modules/@grpc/grpc-js/build/src/client.js:96:33)
    at Http2CallStream.emit (events.js:198:15)
    at process.nextTick (/var/www/html/test/google-ml/node_modules/@grpc/grpc-js/build/src/call-stream.js:79:22)
    at processTicksAndRejections (internal/process/task_queues.js:79:9)
  code: 4,
  details: 'Deadline exceeded',
  metadata: Metadata { internalRepr: Map {}, options: {} } }

The happiness.csv file has the following contents:
gs://example_bucket_ocr_idr2/certs/01.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/02.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/03.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/04.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/05.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/06.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/07.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/08.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/09.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/10.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/11.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/12.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/13.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/14.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/15.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/16.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/17.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/18.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/19.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/20.pdf, certificates, post_graduation
gs://example_bucket_ocr_idr2/certs/21.pdf, certificates, post_graduation

Any help will be appreciated!

@alexander-fenster any thoughts with regard to this issue? Is there a way to extend timeout before DEADLINE_EXCEEDED is fired, or would this need to be broken into more requests (I would expected this to be batched into multiple API requests already).

@soubhikchatterjee One thing to try here would be to increase the timeout manually and see if it helps.  Could you try opening node_modules/@google-cloud/automl/src/v1/auto_ml_client_config.json and, in the following block:
        "ImportData": {
          "timeout_millis": 20000,
          "retry_codes_name": "non_idempotent",
          "retry_params_name": "default"
        },

set the timeout_millis to something more than 20 seconds?  It may or may not help, but let's start from here.

Hello Guys
Thanks for adding replies to this thread, turns out that I was trying to import multi-labeled csv file to a single label dataset. I changed the csv and added just one label in each row.
But this error message is too deceptive. Can this be fixed?

@soubhikchatterjee 👍I'm glad you got things worked out. I'll see if I can pass your feedback on to the appropriate person (about this being confusing, and needing some more documentation).

@alexander-fenster For some files I am still getting the "deadline ..." error, so when i increased the timeout as per your suggestion, it fixed the issue. Is there I can over-ride this value from my nodejs code instead of modifying the files inside the node_modules ?
Many thanks in advance.

I am also getting some DEADLINE_EXCEEDED: Deadline exceeded errors.
I am using "@google-cloud/automl": "^2.5.1",
Here is my code:
`
const client = new automl.v1beta1.PredictionServiceClient()
const modelFullId = client.modelPath(projectId, computeRegion, modelId)
client.predict({
name: modelFullId,
payload: inputs,
params: {feature_importance: false}
})
.then( async responses => {
`

authors: alexander-fenster bcoe bitcoinvsalts soubhikchatterjee

repo: googleapis/nodejs-automl


text:
Improve the Error handling sample and add more documentation

Fixes #83  (it's a good idea to open an issue first for discussion)

[ ] Tests and linter pass
[ ] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate docs were updated (if necessary)

Pull request for the following issue:
https://github.com/googleapis/nodejs-datastore/issues/83

Looks good. So, did we delete this sample file at some point, and now we're bringing it back?

Looks good. So, did we delete this sample file at some point, and now we're bringing it back?

It was deleted some time ago #110.
And now it looks like there is a request to add it back (link)

How can we make sure this new sample will have a home on the docs site and not be axed again?

Perhaps bring it up on the chat and ask Frank Natividad?

cc @frankyn!

Summoning my colleague @benwhitehead, who is my point person for Datastore/Firestore related DevEx.

I'll poke around in docsite and figure out how/where this was published before and circle back.

@googlebot I consent.

authors: AVaksman BenWhitehead frankyn stephenplusplus vishald123

repo: googleapis/nodejs-datastore


text:
Add support for angular

Error message
ERROR in ./node_modules/google-gax/build/src/operationsClient.js
Module not found: Error: Can't resolve './operations_client_config' in 'C:\Users\chenxi.zhao\workspace\chatbot-designer\node_modules\google-gax\build\src'
ERROR in ./node_modules/google-auth-library/build/src/auth/googleauth.js
Module not found: Error: Can't resolve 'child_process' in 'C:\Users\chenxi.zhao\workspace\chatbot-designer\node_modules\google-auth-library\build\src\auth'

Environment details

OS: WIN 10
Node.js version: 10.16.0
npm version: 6.9.0
dialogflow version:0.10.2

Steps to reproduce
I am very sure the library google-gax and google-auth-library is correct installed.
But I got error as above.
Basically, this is angular8 + universal project.
when I build project with  npm run build:ssr, it says the modules not found.
I google a lot, it says google library compact with webpack? I tried everything, but nothing works.
import {Observable} from "rxjs";

import { Intent } from 'src/app/models/intent';

const dialogflow = require('dialogflow');

@Injectable({
providedIn: 'root'
})
export class IntentService {
.......
.......
readonly intentsClient = new dialogflow.IntentsClient(this.opts);
readonly request = {
    parent: this.projectAgentPath,
    intentView: 'INTENT_VIEW_FULL',
};

constructor() { }

fetchIntents(): Observable<Intent[]> {
  return this.intentsClient.listIntents(this.request);
}
}

Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!

👋 hello @ceciliazcx, currently nodejs-dialogflow does not work in a frontend context, as it relies on libraries that exist in Node.js but not the browser, e.g., http2, child_process.
there are efforts in the works to make libraries like dialogflow work for the frontend, but we're not quite there yet.

@bcoe ok. Thx for reply. I basically only want to get intents from dialog flow. Is there any other I can do it from angular?

Could you give this library a try?
https://github.com/googleapis/google-api-nodejs-client/
It's less easy to work with, but is known to be webpack-able.

Going to close this as a duplicate of https://github.com/googleapis/google-cloud-node/issues/2933

Hey guys, is angular still not supported? I am trying to add dialog flow to my ionic angular project and got the same error.

authors: JustinBeckwith bcoe callmehiphop ceciliazcx frederickjjoubert

repo: googleapis/nodejs-dialogflow


text:
chore: disable proto validation by the generator

The generator Docker image will start supporting this option soon.
The command line will be: ..... --validation false .....

The sample test should work with the latest gax to be published.


text:
fix: in Koa2 interface await next as a function

Fixes: #335

@ofrobots I'm in the process of creating the tests, and I think they should be samples tests.  In particular, I am updating the samples to include samples for the different frameworks.  Then I'll update the samples tests to test each sample.

Since the sample test changes involve a bit of work, I made an issue (#337) to track it.  A follow-up PR will implement that work, and thus I will land this PR.  That way, each PR can focus on a specific task.

Repeating my comment on the issue, this fixed the koa interface. Can you fix the koa2 interface, as was described in the 0.6.2 release changelog (and the name of the PR :wink: )?

authors: DominicKramer codyzu

repo: googleapis/nodejs-error-reporting


text:
fix: speed up listDocuments pagination

A page size of 2**32 - 1 is rejected by the backend, making it to use the default page size of 20, which increases the latency of the listDocuments call.
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

[x] Make sure to open an issue as a bug/issue before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
[ ] Ensure the tests and linter pass
[ ] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate docs were updated (if necessary)

Fixes #1342 🦕

No idea what to do about code coverage, as nothing really changed there, only an extra comment line was added.

No idea what to do about code coverage, as nothing really changed there, only an extra comment line was added.

Don't worry about the coverage check. I will take a look at the PR today.

Hi!
Now I'm writing scripts for delete Firestore data.
and I had stack for memory leak.
Because listDocuments() makes 100000 of array data.
I found this issues.
Is it possible to pagination listDocuments()?

@wimpykid719 Are you able to use "Firestore.recursiveDelete()", which deletes documents in smaller batches? If that is not possible, you can use the Query API and combine it with limits or startAt().

Yeah, I want to do that but It's so deep path and I'm stacking the middle of path.
/collection/document/collection/document/collection/document/collection/documents(The id is like this 20210930114547 this is almost random)/collection/documents(random ID)/field
I'm trying to get documents id using listDocuments() for making delete path, but this documents return to so many array that makes memory leak.
and I think can't use Query API and limits(), startAt() because that documents doesn't have field.
Do you have any ideas? Please🙇🏻♂

You can also use https://github.com/googleapis/nodejs-firestore/blob/main/dev/src/v1/firestore_client.ts#L1343 directly, which lets you control the output size.

authors: schmidt-sebastian sk- wimpykid719

repo: googleapis/nodejs-firestore


text:
Synthesis failed for nodejs-notebooks

Hello! Autosynth couldn't regenerate nodejs-notebooks. :broken_heart:
Here's the output from running synth.py:
ntinuous/node12/test.cfg
.kokoro/docs.sh
.kokoro/lint.sh
.kokoro/populate-secrets.sh
.kokoro/presubmit/node10/common.cfg
.kokoro/presubmit/node12/common.cfg
.kokoro/presubmit/node12/samples-test.cfg
.kokoro/presubmit/node12/system-test.cfg
.kokoro/presubmit/node12/test.cfg
.kokoro/publish.sh
.kokoro/release/docs-devsite.cfg
.kokoro/release/docs-devsite.sh
.kokoro/release/docs.cfg
.kokoro/release/docs.sh
.kokoro/release/publish.cfg
.kokoro/samples-test.sh
.kokoro/system-test.sh
.kokoro/test.bat
.kokoro/test.sh
.kokoro/trampoline.sh
.kokoro/trampoline_v2.sh
.mocharc.js
.nycrc
.prettierignore
.prettierrc.js
.trampolinerc
CODE_OF_CONDUCT.md
CONTRIBUTING.md
LICENSE
README.md
api-extractor.json
renovate.json
samples/README.md
2020-11-17 04:47:10,560 synthtool [DEBUG] > Post-processing GAPIC library...
DEBUG:synthtool:Post-processing GAPIC library...
2020-11-17 04:47:10,560 synthtool [DEBUG] > Installing dependencies...
DEBUG:synthtool:Installing dependencies...

> protobufjs@6.10.2 postinstall /home/kbuilder/.cache/synthtool/nodejs-notebooks/node_modules/protobufjs
> node scripts/postinstall

> @google-cloud/notebooks@1.0.0 prepare /home/kbuilder/.cache/synthtool/nodejs-notebooks
> npm run compile-protos && npm run compile

> @google-cloud/notebooks@1.0.0 compile-protos /home/kbuilder/.cache/synthtool/nodejs-notebooks
> compileProtos src

installing semver@^7.1.2
installing tmp@^0.2.0
installing uglify-js@^3.7.7
installing espree@^7.0.0
installing escodegen@^2.0.0
installing estraverse@^5.1.0

> @google-cloud/notebooks@1.0.0 compile /home/kbuilder/.cache/synthtool/nodejs-notebooks
> tsc -p . && cp -r protos build/

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.1.2 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 646 packages from 760 contributors and audited 647 packages in 26.934s

58 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

2020-11-17 04:47:38,138 synthtool [DEBUG] > Running prelint...
DEBUG:synthtool:Running prelint...

> @google-cloud/notebooks@1.0.0 prelint /home/kbuilder/.cache/synthtool/nodejs-notebooks
> cd samples; npm link ../; npm i

> @google-cloud/notebooks@1.0.0 prepare /home/kbuilder/.cache/synthtool/nodejs-notebooks
> npm run compile-protos && npm run compile

> @google-cloud/notebooks@1.0.0 compile-protos /home/kbuilder/.cache/synthtool/nodejs-notebooks
> compileProtos src

> @google-cloud/notebooks@1.0.0 compile /home/kbuilder/.cache/synthtool/nodejs-notebooks
> tsc -p . && cp -r protos build/

npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.3 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

audited 647 packages in 14.138s

58 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

/home/kbuilder/.nvm/versions/node/v14.15.1/lib/node_modules/@google-cloud/notebooks -> /home/kbuilder/.cache/synthtool/nodejs-notebooks
/home/kbuilder/.cache/synthtool/nodejs-notebooks/samples/node_modules/@google-cloud/notebooks -> /home/kbuilder/.nvm/versions/node/v14.15.1/lib/node_modules/@google-cloud/notebooks -> /home/kbuilder/.cache/synthtool/nodejs-notebooks
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.1.2 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 137 packages from 118 contributors and audited 225 packages in 4.792s

58 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

2020-11-17 04:47:58,668 synthtool [DEBUG] > Running fix...
DEBUG:synthtool:Running fix...

> @google-cloud/notebooks@1.0.0 fix /home/kbuilder/.cache/synthtool/nodejs-notebooks
> gts fix

version: 14

/home/kbuilder/.cache/synthtool/nodejs-notebooks/src/v1beta1/notebook_service_client.ts
  105:48  error  'window' is not defined  no-undef

✖ 1 problem (1 error, 0 warnings)

Error: Command failed with exit code 1: node ./node_modules/eslint/bin/eslint --fix **/*.ts **/*.js **/*.tsx **/*.jsx --no-error-on-unmatched-pattern
    at makeError (/home/kbuilder/.cache/synthtool/nodejs-notebooks/node_modules/execa/lib/error.js:59:11)
    at handlePromise (/home/kbuilder/.cache/synthtool/nodejs-notebooks/node_modules/execa/index.js:114:26)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async run (/home/kbuilder/.cache/synthtool/nodejs-notebooks/node_modules/gts/build/src/cli.js:120:17) {
  shortMessage: 'Command failed with exit code 1: node ./node_modules/eslint/bin/eslint --fix **/*.ts **/*.js **/*.tsx **/*.jsx --no-error-on-unmatched-pattern',
  command: 'node ./node_modules/eslint/bin/eslint --fix **/*.ts **/*.js **/*.tsx **/*.jsx --no-error-on-unmatched-pattern',
  exitCode: 1,
  signal: undefined,
  signalDescription: undefined,
  stdout: undefined,
  stderr: undefined,
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @google-cloud/notebooks@1.0.0 fix: `gts fix`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @google-cloud/notebooks@1.0.0 fix script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kbuilder/.npm/_logs/2020-11-17T12_48_17_599Z-debug.log
2020-11-17 04:48:17,607 synthtool [ERROR] > Failed executing npm run fix:

None
ERROR:synthtool:Failed executing npm run fix:

None
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 102, in <module>
    main()
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kbuilder/.cache/synthtool/nodejs-notebooks/synth.py", line 38, in <module>
    node.postprocess_gapic_library()
  File "/tmpfs/src/github/synthtool/synthtool/languages/node.py", line 194, in postprocess_gapic_library
    fix(hide_output=hide_output)
  File "/tmpfs/src/github/synthtool/synthtool/languages/node.py", line 179, in fix
    shell.run(["npm", "run", "fix"], hide_output=hide_output)
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 39, in run
    raise exc
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['npm', 'run', 'fix']' returned non-zero exit status 1.
2020-11-17 04:48:17,679 autosynth [ERROR] > Synthesis failed
2020-11-17 04:48:17,679 autosynth [DEBUG] > Running: git reset --hard HEAD
HEAD is now at 503f306 build(node): add KOKORO_BUILD_ARTIFACTS_SUBDIR to env
2020-11-17 04:48:17,695 autosynth [DEBUG] > Running: git checkout autosynth
Switched to branch 'autosynth'
2020-11-17 04:48:17,713 autosynth [DEBUG] > Running: git clean -fdx
Removing __pycache__/
Removing build/
Removing node_modules/
Removing package-lock.json
Removing samples/node_modules/
Removing samples/package-lock.json
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 354, in <module>
    main()
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 189, in main
    return _inner_main(temp_dir)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 334, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 65, in synthesize_loop
    has_changes = toolbox.synthesize_version_in_new_branch(synthesizer, youngest)
  File "/tmpfs/src/github/synthtool/autosynth/synth_toolbox.py", line 259, in synthesize_version_in_new_branch
    synthesizer.synthesize(synth_log_path, self.environ)
  File "/tmpfs/src/github/synthtool/autosynth/synthesizer.py", line 120, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.

Autosynth is still having trouble generating nodejs-notebooks. :sob:
Here's the output from running synth.py:
/system-test.cfg
.kokoro/continuous/node12/test.cfg
.kokoro/docs.sh
.kokoro/lint.sh
.kokoro/populate-secrets.sh
.kokoro/presubmit/node10/common.cfg
.kokoro/presubmit/node12/common.cfg
.kokoro/presubmit/node12/samples-test.cfg
.kokoro/presubmit/node12/system-test.cfg
.kokoro/presubmit/node12/test.cfg
.kokoro/publish.sh
.kokoro/release/docs-devsite.cfg
.kokoro/release/docs-devsite.sh
.kokoro/release/docs.cfg
.kokoro/release/docs.sh
.kokoro/release/publish.cfg
.kokoro/samples-test.sh
.kokoro/system-test.sh
.kokoro/test.bat
.kokoro/test.sh
.kokoro/trampoline.sh
.kokoro/trampoline_v2.sh
.mocharc.js
.nycrc
.prettierignore
.prettierrc.js
.trampolinerc
CODE_OF_CONDUCT.md
CONTRIBUTING.md
LICENSE
README.md
api-extractor.json
renovate.json
samples/README.md
2020-11-18 04:47:57,829 synthtool [DEBUG] > Post-processing GAPIC library...
DEBUG:synthtool:Post-processing GAPIC library...
2020-11-18 04:47:57,829 synthtool [DEBUG] > Installing dependencies...
DEBUG:synthtool:Installing dependencies...

> protobufjs@6.10.2 postinstall /home/kbuilder/.cache/synthtool/nodejs-notebooks/node_modules/protobufjs
> node scripts/postinstall

> @google-cloud/notebooks@1.0.1 prepare /home/kbuilder/.cache/synthtool/nodejs-notebooks
> npm run compile-protos && npm run compile

> @google-cloud/notebooks@1.0.1 compile-protos /home/kbuilder/.cache/synthtool/nodejs-notebooks
> compileProtos src

installing semver@^7.1.2
installing tmp@^0.2.0
installing uglify-js@^3.7.7
installing espree@^7.0.0
installing escodegen@^2.0.0
installing estraverse@^5.1.0

> @google-cloud/notebooks@1.0.1 compile /home/kbuilder/.cache/synthtool/nodejs-notebooks
> tsc -p . && cp -r protos build/

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.1.2 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 646 packages from 760 contributors and audited 647 packages in 29.397s

58 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

2020-11-18 04:48:27,896 synthtool [DEBUG] > Running prelint...
DEBUG:synthtool:Running prelint...

> @google-cloud/notebooks@1.0.1 prelint /home/kbuilder/.cache/synthtool/nodejs-notebooks
> cd samples; npm link ../; npm i

> @google-cloud/notebooks@1.0.1 prepare /home/kbuilder/.cache/synthtool/nodejs-notebooks
> npm run compile-protos && npm run compile

> @google-cloud/notebooks@1.0.1 compile-protos /home/kbuilder/.cache/synthtool/nodejs-notebooks
> compileProtos src

> @google-cloud/notebooks@1.0.1 compile /home/kbuilder/.cache/synthtool/nodejs-notebooks
> tsc -p . && cp -r protos build/

npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.1.3 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

audited 647 packages in 14.698s

58 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

/home/kbuilder/.nvm/versions/node/v14.15.1/lib/node_modules/@google-cloud/notebooks -> /home/kbuilder/.cache/synthtool/nodejs-notebooks
/home/kbuilder/.cache/synthtool/nodejs-notebooks/samples/node_modules/@google-cloud/notebooks -> /home/kbuilder/.nvm/versions/node/v14.15.1/lib/node_modules/@google-cloud/notebooks -> /home/kbuilder/.cache/synthtool/nodejs-notebooks
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.1.2 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 137 packages from 118 contributors and audited 225 packages in 5.096s

58 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

2020-11-18 04:48:49,305 synthtool [DEBUG] > Running fix...
DEBUG:synthtool:Running fix...

> @google-cloud/notebooks@1.0.1 fix /home/kbuilder/.cache/synthtool/nodejs-notebooks
> gts fix

version: 14

/home/kbuilder/.cache/synthtool/nodejs-notebooks/src/v1beta1/notebook_service_client.ts
  111:48  error  'window' is not defined  no-undef

✖ 1 problem (1 error, 0 warnings)

Error: Command failed with exit code 1: node ./node_modules/eslint/bin/eslint --fix **/*.ts **/*.js **/*.tsx **/*.jsx --no-error-on-unmatched-pattern
    at makeError (/home/kbuilder/.cache/synthtool/nodejs-notebooks/node_modules/execa/lib/error.js:59:11)
    at handlePromise (/home/kbuilder/.cache/synthtool/nodejs-notebooks/node_modules/execa/index.js:114:26)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async run (/home/kbuilder/.cache/synthtool/nodejs-notebooks/node_modules/gts/build/src/cli.js:120:17) {
  shortMessage: 'Command failed with exit code 1: node ./node_modules/eslint/bin/eslint --fix **/*.ts **/*.js **/*.tsx **/*.jsx --no-error-on-unmatched-pattern',
  command: 'node ./node_modules/eslint/bin/eslint --fix **/*.ts **/*.js **/*.tsx **/*.jsx --no-error-on-unmatched-pattern',
  exitCode: 1,
  signal: undefined,
  signalDescription: undefined,
  stdout: undefined,
  stderr: undefined,
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @google-cloud/notebooks@1.0.1 fix: `gts fix`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @google-cloud/notebooks@1.0.1 fix script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kbuilder/.npm/_logs/2020-11-18T12_49_07_812Z-debug.log
2020-11-18 04:49:07,822 synthtool [ERROR] > Failed executing npm run fix:

None
ERROR:synthtool:Failed executing npm run fix:

None
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 102, in <module>
    main()
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmpfs/src/github/synthtool/env/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/tmpfs/src/github/synthtool/synthtool/__main__.py", line 94, in main
    spec.loader.exec_module(synth_module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kbuilder/.cache/synthtool/nodejs-notebooks/synth.py", line 38, in <module>
    node.postprocess_gapic_library()
  File "/tmpfs/src/github/synthtool/synthtool/languages/node.py", line 194, in postprocess_gapic_library
    fix(hide_output=hide_output)
  File "/tmpfs/src/github/synthtool/synthtool/languages/node.py", line 179, in fix
    shell.run(["npm", "run", "fix"], hide_output=hide_output)
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 39, in run
    raise exc
  File "/tmpfs/src/github/synthtool/synthtool/shell.py", line 33, in run
    encoding="utf-8",
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['npm', 'run', 'fix']' returned non-zero exit status 1.
2020-11-18 04:49:07,910 autosynth [ERROR] > Synthesis failed
2020-11-18 04:49:07,911 autosynth [DEBUG] > Running: git reset --hard HEAD
HEAD is now at 033c9ce chore: release 1.0.1 (#18)
2020-11-18 04:49:07,928 autosynth [DEBUG] > Running: git checkout autosynth
Switched to branch 'autosynth'
2020-11-18 04:49:07,947 autosynth [DEBUG] > Running: git clean -fdx
Removing __pycache__/
Removing build/
Removing node_modules/
Removing package-lock.json
Removing samples/node_modules/
Removing samples/package-lock.json
Traceback (most recent call last):
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 354, in <module>
    main()
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 189, in main
    return _inner_main(temp_dir)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 334, in _inner_main
    commit_count = synthesize_loop(x, multiple_prs, change_pusher, synthesizer)
  File "/tmpfs/src/github/synthtool/autosynth/synth.py", line 65, in synthesize_loop
    has_changes = toolbox.synthesize_version_in_new_branch(synthesizer, youngest)
  File "/tmpfs/src/github/synthtool/autosynth/synth_toolbox.py", line 259, in synthesize_version_in_new_branch
    synthesizer.synthesize(synth_log_path, self.environ)
  File "/tmpfs/src/github/synthtool/autosynth/synthesizer.py", line 120, in synthesize
    synth_proc.check_returncode()  # Raise an exception.
  File "/home/kbuilder/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 389, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['/tmpfs/src/github/synthtool/env/bin/python3', '-m', 'synthtool', '--metadata', 'synth.metadata', 'synth.py', '--']' returned non-zero exit status 1.

Google internal developers can see the full log here.

Autosynth passed, closing! :green_heart:

repo: googleapis/nodejs-notebooks


text:
table.delete may be confusing for users

Table.prototype.delete results in a DROP TABLE operation on the database.
Here is a quick overview of the surface:
db.table('blah').insert //Inserts a Row
db.table('blah').update //Updates a Row
db.table('blah').upsert //Upsert a Row
db.table('blah').delete //Deletes the table
db.table('blah').deleteRows //Deletes a Row

It may be worth considering moving the functionality of delete to a new name, such as deleteTable, drop, or dropTable to make it clear this isn't an operation to delete rows but the table. Other non qualified operations (insert, update, etc) are on the row so it is odd to have delete be table level.

Also, Database.prototype.delete will delete the database, so we are consistent with this. On Database, createTable is the operation to make a table.
I propose the following
Table.delete => Table.drop
Database.delete => Database.drop
Another client, Python uses the terminology drop as it matches the underlying operation. This would avoid confusion with row deletion methods.

Could we introduce a rows argument to delete that would alter the meaning?
table.delete(rows) = "delete rows in this table"

What if the rows array is empty - would that evaluate differently from table.delete()? Either way, this seems kind of confusing and risky.
I can see the point about it being consistent with other libraries / services, but on the other hand, from the standpoint of most people who work with databases, I think drop() is a little more conventional? The risk of deleting a table when you're trying to delete a row seems important, while it's a mistake one would probably only make once, it could be a pretty unpleasant mistake to make.
Personally, I'd rather see deleteTable to match deleteRows() than to have delete() work both ways....

What if the rows array is empty - would that evaluate differently from table.delete()? Either way, this seems kind of confusing and risky.

My first thought would be, that should throw an error.

but on the other hand, from the standpoint of most people who work with databases, I think drop() is a little more conventional?

Agree-- we should first be conventional with these types of users, rather than general users of our other libraries. I think we could still add the drop() method to support the familiar language.

Personally, I'd rather see deleteTable to match deleteRows() than to have delete() work both ways....

My issue here is the backwards hierarchy:
database -> table -> deleteTable()

On the other hand, deleteTable() could exist on the database instance:
database -> deleteTable()

The risk of deleting a table when you're trying to delete a row seems important

I believe this can be solved by reading the input. But before I'm sure, I want to make sure I understand-- if a user doesn't provide an argument to table.delete(), what would they expect to happen?

I agree with adding a .drop method.
This is the standard vocabulary used in other relational databases and provides a clear delineation between the DML DELETE statements and DDL DROP statements.
Additionally, this has been part of the SQL standard for several decades.
Is there a reason Google is not modeling Spanner SQL syntax from this?

One reason to incorporate @stephenplusplus recommendation is it would allow for the existing code to continue to function as intended, while guarding against people who call delete(rows) meaning to call deleteRows(rows).
If this work is done it will involve deprecating a method and adding a new one and in a future release removing that method. So there could be value in guarding the delete call in the meantime.

Just to document the outcomes of this discussion clearly - we are going to:

Throw an exception if you pass any arguments to the table.delete method.  This means that if a user got confused and tried to use this method to delete rows, we will set them straight.
Add a table.drop method that aliases the table.delete method.

Good idea, on multiple occasions I was just an inch away from deleting an entire table when I was actually trying to delete rows 😅

authors: JustinBeckwith WaldoJeffers crwilcox dsquier stephenplusplus wyardley

repo: googleapis/nodejs-spanner


text:
Faster INSERT methods

Is your feature request related to a problem? Please describe.
I need to work with large amounts of rows on an ETL service and wanted to use SQLAlchemy ORM backed by BigQuery dialect to write more pythonic code, but it is extremely slow to upload more than tens of rows.
Describe the solution you'd like
Be able to use ORM-Enabled INSERT like the SQLAlchemy docs suggest, or at least the soon-to-be legacy Session.bulk_insert_mappings and have better performance
Describe alternatives you've considered
I've tried using all methods provided by this SQLA FAQ and none could be faster than about 900 seconds for a minimum of 200 rows.
Aditional context
While the examples shows that methods like the bulk_insert_mappings could do 0.4 seconds for hundreds of thousands of rows, using BigQuery dialect I've faced 2 h 30min to upload about 4k rows. I don't expect that BQ could reach anything close to the 0.4 seconds for 100k rows, but something around 5, 6 minutes I do believe is possible, since it is the average time the BigQuery client from google.cloud.bigquery can do for an upload of about 90k rows.

This is the result from a test run using the referenced FAQ
Python: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0]
sqlalchemy v1.4.27 (future=True)

SQLA ORM:
          Total time for 10 records 44.846 secs

SQLA ORM pk given:
          Total time for 10 records 37.176 secs

SQLA ORM-Enabled INSERT:
          Total time for 10 records 42.995 secs

SQLA ORM bulk_save_objects:
          Total time for 10 records 22.663 secs

SQLA ORM bulk_save_objects, return_defaults:
          Total time for 10 records 48.917 secs

SQLA ORM bulk_insert_mappings:
          Total time for 10 records 48.129 secs

SQLA ORM bulk_insert_mappings, return_defaults:
          Total time for 10 records 22.078 secs

SQLA Core:
          Total time for 10 records 48.751 secs

BQ Client:
          Total time for 10 records 0.727 secs

This is the code used to test the different possibilities
# -*- coding: utf-8 -*-

from datetime import datetime
from pprint import pprint
from typing import Dict, List
from uuid import uuid4
import contextlib
import time
import sys

from google.cloud.bigquery import Client, Table
from google.api_core import retry

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session
from sqlalchemy import __version__, Column, DateTime, String, create_engine, insert

def make_str_id() -> str:
    return uuid4().hex

Base = declarative_base()

class Customer(Base):
    __tablename__ = "customer"

    id: str = Column('id', String, primary_key=True, default=make_str_id)
    name: str = Column('name', String)
    created_at: datetime = Column('created_at', DateTime, default=datetime.now)

    __mapper_args__ = {
        "version_id_col": created_at,
        "version_id_generator": datetime.now,
        "eager_defaults": True
    }

@contextlib.contextmanager
def sqlalchemy_session(project: str, future: bool):
    uri = f'bigquery://{project}/sandBox'
    engine = create_engine(uri, future=True)
    Base.metadata.create_all(engine, checkfirst=True)
    session = Session(
        bind=engine, future=future,
        autoflush=False, expire_on_commit=False
    )
    yield session
    session.close()
    Base.metadata.drop_all(engine, checkfirst=True)
    engine.dispose()

def print_result(name, nrows, seconds):
    print(f"{name}:\n{' '*10}Total time for {nrows} records {seconds:.3f} secs\n")

def test_sqlalchemy_orm(project: str, n=100000, future=True):
    with sqlalchemy_session(project, future) as session:
        t0 = time.time()
        for i in range(n):
            customer = Customer()
            customer.name = "NAME " + str(i)
            session.add(customer)
            if i % 1000 == 0:
                session.flush()
        session.commit()
        print_result("SQLA ORM", n, time.time() - t0)

def test_sqlalchemy_orm_pk_given(project: str, n=100000, future=True):
    with sqlalchemy_session(project, future) as session:
        t0 = time.time()
        for i in range(n):
            customer = Customer(id=make_str_id(), name="NAME " + str(i))
            session.add(customer)
            if i % 1000 == 0:
                session.flush()
        session.commit()
        print_result("SQLA ORM pk given", n, time.time() - t0)

def test_sqlalchemy_orm_enabled_insert(project: str, n=100000, future=True):
    with sqlalchemy_session(project, future) as session:
        t0 = time.time()
        session.execute(
            insert(Customer),
            [
                {"name": f"NAME{i}"}
                for i in range(n)
            ]
        )
        session.commit()
        print_result("SQLA ORM-Enabled INSERT", n, time.time() - t0)

def test_sqlalchemy_orm_bulk_save_objects(project: str, n=100000, future=True, return_defaults=False):
    with sqlalchemy_session(project, future) as session:
        t0 = time.time()
        for chunk in range(0, n, 10000):
            session.bulk_save_objects(
                [
                    Customer(name="NAME " + str(i))
                    for i in range(chunk, min(chunk + 10000, n))
                ],
                return_defaults=return_defaults,
            )
        session.commit()
        print_result(
            f"SQLA ORM bulk_save_objects{', return_defaults' if return_defaults else ''}",
            n,
            time.time() - t0,
        )

def test_sqlalchemy_orm_bulk_insert(project: str, n=100000, future=True, return_defaults=False):
    with sqlalchemy_session(project, future) as session:
        t0 = time.time()
        for chunk in range(0, n, 10000):
            session.bulk_insert_mappings(
                Customer,
                [
                    dict(name="NAME " + str(i))
                    for i in range(chunk, min(chunk + 10000, n))
                ],
                return_defaults=return_defaults,
            )
        session.commit()
        print_result(
            f"SQLA ORM bulk_insert_mappings{', return_defaults' if return_defaults else ''}",
            n,
            time.time() - t0,
        )

def test_sqlalchemy_core(project: str, n=100000, future=True):
    with sqlalchemy_session(project, future) as session:
        with session.bind.begin() as conn:
            t0 = time.time()
            conn.execute(
                insert(Customer.__table__),
                [{"name": "NAME " + str(i)} for i in range(n)],
            )
            conn.commit()
            print_result("SQLA Core", n, time.time() - t0)

@contextlib.contextmanager
def bigquery_client(project: str) -> Client:
    schema = [
        {"name": "id",
         "type": "STRING",
         "mode":"REQUIRED"},
        {"name": "name",
         "type": "STRING",
         "mode":"NULLABLE"},
        {"name": "created_at",
         "type": "DATETIME",
         "mode":"NULLABLE"},
    ]
    table_path = f"{project}.sandBox.{Customer.__tablename__}"
    with Client(project) as client:
        customer = Table(table_path, schema)
        client.create_table(customer, exists_ok=True)
        yield client
        client.delete_table(table_path)

def test_bigquery(project: str, n=100000):
    retry_ = retry.Retry((lambda e: True), 0.2, 10., 5, 10.)
    with bigquery_client(project) as client:
        client: Client
        t0 = time.time()
        rows = [
            {"id": make_str_id(),
             "name": "NAME " + str(i),
             "created_at": datetime.now()}
            for i in range(n)
        ]
        table_path = f"{project}.sandBox.{Customer.__tablename__}"
        customer = client.get_table(table_path)
        client.insert_rows(customer, rows, retry=retry_)
        print_result("BQ Client", n, time.time() - t0)

if __name__ == "__main__":
    rows = 10
    _future = True
    project_id = 'YOUR_PROJECT_ID'

    print(f"Python: {' '.join(sys.version.splitlines())}")
    print(f"sqlalchemy v{__version__} (future={_future})\n")

    test_sqlalchemy_orm(project_id, rows, _future)
    test_sqlalchemy_orm_pk_given(project_id, rows, _future)
    test_sqlalchemy_orm_enabled_insert(project_id, rows, _future)
    test_sqlalchemy_orm_bulk_save_objects(project_id, rows, _future)
    test_sqlalchemy_orm_bulk_save_objects(project_id, rows, _future, True)
    test_sqlalchemy_orm_bulk_insert(project_id, rows, _future)
    test_sqlalchemy_orm_bulk_insert(project_id, rows, _future, True)
    test_sqlalchemy_core(project_id, rows, _future)
    test_bigquery(project_id, rows)

I am in the same boat where I have to operate on large datasets and use SQLalchemy to format my BigQuery queries.  Since I also use pandas when locally operating on my data, I find I have to use a mixture of these functions and modules for the best performance:
Upload:

google-cloud-bigquery -> load_table_from_dataframe (uses parquet to upload)
pandas-gbq -> to_gbq (uses parquet to upload)

Download:

pandas-gbq -> read_gbq (uses "use_bqstorage_api" argument for fast download)

So it appears all these methods (mostly) bypass using raw SQL data streams and instead convert data to compressed binary files for bulk data transfer.  It would be convenient if all this could be accomplished via python-bigquery-sqlalchemy, but I am not sure if SQLalchemy supports underlying dialects using their own "conversion engines" (lets call it that).
I do see in the documentation for this library, there is a way to supply your own custom BQ client:
https://googleapis.dev/python/sqlalchemy-bigquery/latest/README.html#supplying-your-own-bigquery-client
Maybe there is a way to configure this client to use the faster interact methods/clients to interact with BiqQuery?

BQ's high throughput ingestion is entirely divorced from the query engine, so mapping it into sqlalchemy already feels unusual.  There's work unrelated to sqlalchemy to improve ingestion, but that will take time to propagate into other consumers like this project and other connectors/tools if it makes sense.

BQ's high throughput ingestion is entirely divorced from the query engine, so mapping it into sqlalchemy already feels unusual. There's work unrelated to sqlalchemy to improve ingestion, but that will take time to propagate into other consumers like this project and other connectors/tools if it makes sense.

What do the first two sentences mean, and what is the logical connection between them?
I am also trying to understand why the throughput is so low.  With such bad performance, this makes the library completely non-viable for most use cases.

The BigQuery Write API supports high throughput ingestion, but it is not exposed as part of the SQL query surface.  This is a case of SQLAlchemy not being the right tool for an ingestion-based workload into BigQuery.

I am gonna close this ticket, due to workloads, personpower, and priorities.
There are tools available that enable high throughput ingestion.

@chalmerlowe,
Do you know how hard it would be to implement the dialect options "use_insertmanyvalues" and "use_insertmanyvalues_wo_returning", which were added in SQLalchemy 2.0 to give native ORM session.add() and session.add_all() support for bulk inserting? I tired manually adding it to my environment and I seem to be getting close to enabling support:
#SA ORM BQ bulk insert patch
from sqlalchemy_bigquery.base import BigQueryDialect
BigQueryDialect.use_insertmanyvalues = True
BigQueryDialect.use_insertmanyvalues_wo_returning = True

Unfortunately I seem to be having problems with parameter binding formatting in the google/cloud/bigquery/dbapi/cursor._format_operation_dict function.
I think this would be a great feature for sqlalchemy-bigquery users to get ORM nested object bulk insert support (and non-nested objects for that matter), without having to use the legacy bulk_insert_mapping and buld_save_objects methods that do not support ORM event hooks and do not support nested ORM objects.
Links:
SQLalchemy 2.0 bulk insert support: https://docs.sqlalchemy.org/en/20/changelog/whatsnew_20.html#optimized-orm-bulk-insert-now-implemented-for-all-backends-other-than-mysql
SQLalchemy commit enabling the support: https://github.com/sqlalchemy/sqlalchemy/commit/2bcc97da424eef7db9a5d02f81d0234492541

@jlynchMicron
Thanks for reaching out.
I appreciate your time and effort.
I will do my best to take a look.
One of the issues with taking on new features is that we need to figure out how to integrate those features into the overall package:

is the new code complete and comprehensive and does it cover a reasonable subset of edge-cases
does the new code meet style guidelines and readability, etc
does the new code need unit tests
does the new code need system tests
do all the features in the new code work in all the versions of Python we support or do we need to remove newer language features OR put in workarounds to ensure backward compatibility.

Not saying that this code doesn't meet guidelines, etc. I have not had a chance to take a good look at it.
You prolly already know this, I just want to ensure that you understand why something that seems as simple as add this snippet of new code takes time and effort to do, especially balanced against my existing TODO list.
I hope to be able to sit and look at this in depth, soon.
Will keep you posted and will reopen this issue.

Hi @chalmerlowe
Yeah, I completely understand all the overhead that's involved with onboarding new code features.  I figured I would share my patch that accomplished my needs and could be the blueprints to solve other people's issues as well.  Thanks!

@chalmerlowe, sounds good and thanks!  Yeah the DML batching patch should arguably be changed at the python bigquery cursor level (this ticket https://github.com/googleapis/python-bigquery/issues/2048 ), but I do think it would be nice to be able to select an insert "style" at the bigquery sqlalchemy level.

authors: JoaoPaes-at-Dynamox chalmerlowe firezdog jlynchMicron shollyman

repo: googleapis/python-bigquery-sqlalchemy

license: mit


text:
Dataflow data pipeline support

Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
Is your feature request related to a problem? Please describe.
Dataflow data pipeline is not supported, and also not available in gcloud CLI (but this is another question).
Describe the solution you'd like
Just add the feature support to the SDK.
Describe alternatives you've considered
REST API.
Additional context
None

This feature request is for a client library for datapipelines.googleapis.com.

I'm going to transfer this issue to the google-cloud-python repository as this is the new home for new python client library feature requests.

repo: googleapis/python-dataflow-client


text:
Dialogflow client does not use HTTP_PROXY and HTTPS_PROXY system variables

OS: Windows 10 Pro
Technology : Python
Module: Dialogflow
Module version: 0.5.2
I used open internet to test a code that is similar to this code:
https://github.com/googleapis/dialogflow-python-client-v2/blob/master/samples/detect_intent_texts.py
It worked fine.
I then had to use proxy, so I set HTTP_PROXY and HTTPS_PROXY. APIAI python client works fine, but Dialogflow python client does not work.
Got this error:
Traceback (most recent call last):
  File "\AppData\Local\Programs\Python\Python37\lib\site-packages\google\api_core\grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "\AppData\Local\Programs\Python\Python37\lib\site-packages\grpc\_channel.py", line 549, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "\AppData\Local\Programs\Python\Python37\lib\site-packages\grpc\_channel.py", line 466, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1557313328.773000000","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2267,"referenced_errors":[{"created":"@1557313328.773000000","description":"Pick Cancelled","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":221,"referenced_errors":[{"created":"@1557313328.773000000","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}]}]}"

This may be related to #34

This issue may no longer be relevant. I am now closing this issue.

authors: JeevaTM b-loved-dreamer busunkim96

repo: googleapis/python-dialogflow


text:
strange interaction with autorelease: snooze and the GitHub app

I think we're seeing weirdness with the snooze functionality because the GitHub app is marking the pull request as autorelease: closed when the PR is closed (even with snoozing).
Merged release PR without autorelease: pending label
Closed, snoozed PR
updated code for 741, but update requested for 713

Then it the library tries to reopen the snoozed PR and there is an existing PR open for that branch so validation fails
Validation Failed: {"resource":"PullRequest","code":"custom","field":"state","message":"state cannot be changed. There is already an open pull request from release-please--branches--master to master."}

This issue will fall out of SLO in 6 days.

repo: googleapis/release-please


text:
feat: parse footer for release-as, which should take precedence

to allow a bump from 0.x.x to 1.0.0, let's allow folks to provide a Relese-As: footer, it would look something like this:
feat: we are going ga \o/

Release-As: 1.0.0

What would the semantics be if there are multiple commits with different Release-As notations?

perhaps take the most chronologically recent?

authors: bcoe dazuma


text:
Open House timezone inconsistency

Hi. I just noticed that all of the slides you uploaded say that the next Open House meeting happens at "5pm CET" while the README says "5pm CEST". It looks like I missed today's Open House for this reason... It would be cool to fix this inconsistency, at least for the next slides! :)

Yeah, I agree that this CET/CEST difference is quite a pain. :)
Thanks a lot!

@pifon2a @gaschler So we don't forget :)

authors: FlorentRevest cschuet

repo: googlecartographer/cartographer


text:
How to use imu in 2d cartographer

Hi,
when I use cartographer without imu,everything is OK. But when I add imu ,no map output，but have pointclouds,look like this:

this is lua and launch:
include "map_builder.lua"
include "trajectory_builder.lua"

options = {
  map_builder = MAP_BUILDER,
  trajectory_builder = TRAJECTORY_BUILDER,
  map_frame = "map",
  tracking_frame = "base_link",
  published_frame = "base_link",
  odom_frame = "odom",
  provide_odom_frame = true,
  publish_frame_projected_to_2d = false,
  use_pose_extrapolator = true,
  use_odometry = false,
  use_nav_sat = false,
  use_landmarks = false,
  num_laser_scans = 1,
  num_multi_echo_laser_scans = 0,
  num_subdivisions_per_laser_scan = 1,
  num_point_clouds = 0,
  lookup_transform_timeout_sec = 0.2,
  submap_publish_period_sec = 0.3,
  pose_publish_period_sec = 5e-3,
  trajectory_publish_period_sec = 30e-3,
  rangefinder_sampling_ratio = 1.,
  odometry_sampling_ratio = 1.,
  fixed_frame_pose_sampling_ratio = 1.,
  imu_sampling_ratio = 1.,
  landmarks_sampling_ratio = 1.,
}

MAP_BUILDER.use_trajectory_builder_2d = true

TRAJECTORY_BUILDER_2D.submaps.num_range_data = 35
TRAJECTORY_BUILDER_2D.min_range = 0.1
TRAJECTORY_BUILDER_2D.max_range = 30.
TRAJECTORY_BUILDER_2D.missing_data_ray_length = 1.
TRAJECTORY_BUILDER_2D.use_imu_data = true
TRAJECTORY_BUILDER_2D.use_online_correlative_scan_matching = true
TRAJECTORY_BUILDER_2D.real_time_correlative_scan_matcher.linear_search_window = 0.1
TRAJECTORY_BUILDER_2D.real_time_correlative_scan_matcher.translation_delta_cost_weight = 10.
TRAJECTORY_BUILDER_2D.real_time_correlative_scan_matcher.rotation_delta_cost_weight = 1e-1

POSE_GRAPH.optimization_problem.huber_scale = 1e2
POSE_GRAPH.optimize_every_n_nodes = 0
POSE_GRAPH.constraint_builder.min_score = 0.7

return options

<launch>
  <arg name="device" default="/dev/ttyACM1"/>
  <arg name="with_plot" default="false"/>
  <node name="adxl345" pkg="adi_driver" type="adxl345_node">
    <param name="device" value="$(arg device)"/>
  </node>

  <node name="cartographer_node" pkg="cartographer_ros"
      type="cartographer_node" args="
          -configuration_directory $(find cartographer_ros)/configuration_files
          -configuration_basename hokuyo.lua"
     output="screen">
      <remap from="scan" to="scan" />
      <remap from="imu" to="imu/data_raw" />

 <node name="cartographer_occupancy_grid_node" pkg="cartographer_ros"
      type="cartographer_occupancy_grid_node" args="-resolution 0.15" />
 <node name="rviz" pkg="rviz" type="rviz" required="true"
      args="-d $(find cartographer_ros)/configuration_files/demo_2d.rviz" />
</launch>

Please check the terminal output, there will almost certainly be errors there, about the topic matching or frame ids regarding IMU.

[ INFO] [1539594822.504915370]: I1015 17:13:42.000000  4085 motion_filter.cc:42] Motion filter reduced the number of nodes to 1.66667%.

If you want, you could turn off the motion filter so more submaps are created even though the lidar doesn't move, so occupancy grids will be output earlier.
Also, please run cartographer_rosbag_validate. IMU data can easily be formatted wrong.

@gaschler  Thank you very much.


text:
[WIP] Load basic mapping information from .designspace files

TODO: reconstruct on top of #56, #57
Build on both #46 and #47 which is hard to express in GH. Please prefer to review the actual linear map in #46.
Step toward #22.

I believe everything interesting here has been merged in other PRs; closing

repo: googlefonts/fontmake-rs


text:
Build fails

I've followed the build instructions from BUILD.md. The following step fails:
(cd colrv1 && rm -rf build/ && time nanoemoji *.toml)

with multiple errors like this:
FAILED: picosvg/clipped/emoji_u1f4f0.svg
picosvg  --clip_to_viewbox --output_file picosvg/clipped/emoji_u1f4f0.svg ../../svg/emoji_u1f4f0.svg
Traceback (most recent call last):
  File "/home/behdad/noto-emoji/venv/bin/picosvg", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/picosvg/picosvg.py", line 76, in main
    app.run(_run, argv=argv)
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
             ^^^^^^^^^^
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/picosvg/picosvg.py", line 58, in _run
    svg = svg.topicosvg(
          ^^^^^^^^^^^^^^
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/picosvg/svg.py", line 1337, in topicosvg
    svg.topicosvg(
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/picosvg/svg.py", line 1361, in topicosvg
    self.simplify(inplace=True)
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/picosvg/svg.py", line 813, in simplify
    self._simplify()
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/picosvg/svg.py", line 745, in _simplify
    paths = list(self._stroke(paths[0]))
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/picosvg/svg.py", line 825, in _stroke
    stroke = shape.as_path().update_path(shape.stroke_commands(self.tolerance))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/picosvg/svg_types.py", line 696, in update_path
    for cmd, args in svg_cmds:
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/picosvg/svg_pathops.py", line 101, in svg_commands
    for svg_cmd, svg_args in _SKIA_CMD_TO_SVG_CMD[cmd](points):
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/picosvg/svg_pathops.py", line 55, in _qcurveto_to_svg
    yield ("Q", control_pt + end_pt)
                ~~~~~~~~~~~^~~~~~~~
TypeError: can only concatenate tuple (not "NoneType") to tuple
[3031/7445] /home/behdad/noto-emoji...on picosvg/clipped/emoji_u1f4e0.svg
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/behdad/noto-emoji/venv/bin/nanoemoji", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/nanoemoji/nanoemoji.py", line 761, in main
    app.run(_run)
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
             ^^^^^^^^^^
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/nanoemoji/nanoemoji.py", line 756, in _run
    maybe_run_ninja(build_file)
  File "/home/behdad/noto-emoji/venv/lib64/python3.11/site-packages/nanoemoji/ninja.py", line 125, in maybe_run_ninja
    subprocess.run(ninja_cmd, check=True)
  File "/usr/lib64/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-C', '/home/behdad/noto-emoji/colrv1/build']' returned non-zero exit status 1.

==
Another minor issue, with the build instructions:
# If you are updating to a new Unicode rev, update configs
(cd colrv1 && python colrv1_generate_configs.py)
git diff colrv1/*.toml

This doesn't work. The colrv1_generate_configs.py is in the repo root, not in the colrv1 directory.

Let's keep it open in case others stumble on this, until I fix the issue in picosvg

this should be now fixed with the latest picosvg 0.22.1
https://github.com/googlefonts/picosvg/releases/tag/v0.22.1

I trust you it works. :)

authors: anthrotype behdad

repo: googlefonts/noto-emoji


text:
Update glyphsLib requirement

Hey folks,
I've updated the pip requirements so fontmake now uses glyphLib's pr googlei18n/glyphsLib@43e199d99e7caaad0d8f3c5b1751ce4ade9a0b89
Thanks,
Marc

did you sign the CLA?

Yep, all signed up.
Thanks again Cosimo.

Hey, I can't merge this yet because git thinks there are conflicts with other recent updates to requirements.txt. Could you resolve and push?

Hi James,
I'll close this and submit another pr.
Thanks

authors: anthrotype jamesgk m4rc1e