text:
Generate external transform wrappers using a script

Implementing a script that generates wrappers for external SchemaTransforms, according to Option #3 in the following design doc: https://s.apache.org/autogen-wrappers
The script's workflow takes place in setup.py, which can be invoked for local setup and creating artifacts for releases. Generated files are suffixed with _et.py so that we can easily tell Git to ignore them and so we can easily delete during cleanup.
Wrappers are generated along with any documentation provided from the underlying SchemaTransform. Modules are generated in compliance with our existing linting rules.
Including documentation for how to use the script.
Including unit tests for different parts of the script.
With the expansion service YAML config, the following transform YAML config is generated:

 Transform YAML config
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# NOTE: This file is autogenerated and should not be edited by hand.

- default_service: sdks:java:io:expansion-service:shadowJar
  description: 'Outputs a PCollection of Beam Rows, each containing a single INT64
    number called "value". The count is produced from the given "start"value and either
    up to the given "end" or until 2^63 - 1.

    To produce an unbounded PCollection, simply do not specify an "end" value. Unbounded
    sequences can specify a "rate" for output elements.

    In all cases, the sequence of numbers is generated in parallel, so there is no
    inherent ordering between the generated values'
  destinations:
    python: apache_beam/io/generate_sequence
  fields:
    end:
      description: The maximum number to generate (exclusive). Will be an unbounded
        sequence if left unspecified.
      nullable: true
      type: numpy.int64
    rate:
      description: Specifies the rate to generate a given number of elements per a
        given number of seconds. Applicable only to unbounded sequences.
      nullable: true
      type: Row(elements=<class 'numpy.int64'>, seconds=typing.Union[numpy.int64,
        NoneType])
    start:
      description: The minimum number to generate (inclusive).
      nullable: false
      type: numpy.int64
  identifier: beam:schematransform:org.apache.beam:generate_sequence:v1
  name: GenerateSequence
- default_service: sdks:java:io:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/kafka
  fields:
    autoOffsetResetConfig:
      description: "What to do when there is no initial offset in Kafka or if the\
        \ current offset does not exist any more on the server. (1) earliest: automatically\
        \ reset the offset to the earliest offset. (2) latest: automatically reset\
        \ the offset to the latest offset (3) none: throw exception to the consumer\
        \ if no previous offset is found for the consumer\u2019s group"
      nullable: true
      type: str
    bootstrapServers:
      description: "A list of host/port pairs to use for establishing the initial\
        \ connection to the Kafka cluster. The client will make use of all servers\
        \ irrespective of which servers are specified here for bootstrapping\u2014\
        this list only impacts the initial hosts used to discover the full set of\
        \ servers. This list should be in the form `host1:port1,host2:port2,...`"
      nullable: false
      type: str
    confluentSchemaRegistrySubject:
      description: ''
      nullable: true
      type: str
    confluentSchemaRegistryUrl:
      description: ''
      nullable: true
      type: str
    consumerConfigUpdates:
      description: 'A list of key-value pairs that act as configuration parameters
        for Kafka consumers. Most of these configurations will not be needed, but
        if you need to customize your Kafka consumer, you may use this. See a detailed
        list: https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html'
      nullable: true
      type: Mapping[str, str]
    errorHandling:
      description: This option specifies whether and where to output unwritable rows.
      nullable: true
      type: Row(output=<class 'str'>)
    fileDescriptorPath:
      description: The path to the Protocol Buffer File Descriptor Set file. This
        file is used for schema definition and message serialization.
      nullable: true
      type: str
    format:
      description: 'The encoding format for the data stored in Kafka. Valid options
        are: RAW,AVRO,JSON,PROTO'
      nullable: true
      type: str
    messageName:
      description: The name of the Protocol Buffer message to be used for schema extraction
        and data conversion.
      nullable: true
      type: str
    schema:
      description: The schema in which the data is encoded in the Kafka topic. For
        AVRO data, this is a schema defined with AVRO schema syntax (https://avro.apache.org/docs/1.10.2/spec.html#schemas).
        For JSON data, this is a schema defined with JSON-schema syntax (https://json-schema.org/).
        If a URL to Confluent Schema Registry is provided, then this field is ignored,
        and the schema is fetched from Confluent Schema Registry.
      nullable: true
      type: str
    topic:
      description: ''
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:kafka_read:v1
  name: ReadFromKafka
- default_service: sdks:java:io:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/kafka
  fields:
    bootstrapServers:
      description: "A list of host/port pairs to use for establishing the initial\
        \ connection to the Kafka cluster. The client will make use of all servers\
        \ irrespective of which servers are specified here for bootstrapping\u2014\
        this list only impacts the initial hosts used to discover the full set of\
        \ servers. | Format: host1:port1,host2:port2,..."
      nullable: false
      type: str
    fileDescriptorPath:
      description: The path to the Protocol Buffer File Descriptor Set file. This
        file is used for schema definition and message serialization.
      nullable: true
      type: str
    format:
      description: 'The encoding format for the data stored in Kafka. Valid options
        are: RAW,JSON,AVRO,PROTO'
      nullable: false
      type: str
    messageName:
      description: The name of the Protocol Buffer message to be used for schema extraction
        and data conversion.
      nullable: true
      type: str
    producerConfigUpdates:
      description: 'A list of key-value pairs that act as configuration parameters
        for Kafka producers. Most of these configurations will not be needed, but
        if you need to customize your Kafka producer, you may use this. See a detailed
        list: https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html'
      nullable: true
      type: Mapping[str, str]
    topic:
      description: ''
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:kafka_write:v1
  name: WriteToKafka
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/json_write
  fields:
    path:
      description: The file path to write to.
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:json_write:v1
  name: JsonWrite
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/bigquery
  fields:
    createDisposition:
      description: ''
      nullable: false
      type: str
    tableSpec:
      description: ''
      nullable: false
      type: str
    writeDisposition:
      description: ''
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:bigquery_fileloads_write:v1
  name: FileLoadsToBigQuery
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/pubsub_read
  fields:
    attributes:
      description: Any additional pubsub attributes that should be populated as String
        fields in the ouptut rows.
      nullable: true
      type: Sequence[str]
    attributesMap:
      description: Any additional field that should be populated with the full set
        of PubSub attributes.
      nullable: true
      type: str
    clientFactory:
      description: ''
      nullable: true
      type: Row()
    clock:
      description: ''
      nullable: true
      type: Row()
    errorHandling:
      description: Specifies how to handle errors.
      nullable: true
      type: Row(output=<class 'str'>)
    format:
      description: 'The encoding format for the data stored in Pubsub. Valid options
        are: RAW,AVRO,JSON'
      nullable: false
      type: str
    idAttribute:
      description: When reading from Cloud Pub/Sub where unique record identifiers
        are provided as Pub/Sub message attributes, specifies the name of the attribute
        containing the unique identifier. The value of the attribute can be any string
        that uniquely identifies this record. Pub/Sub cannot guarantee that no duplicate
        data will be delivered on the Pub/Sub stream. If idAttribute is not provided,
        Beam cannot guarantee that no duplicate data will be delivered, and deduplication
        of the stream will be strictly best effort.
      nullable: true
      type: str
    schema:
      description: The schema in which the data is encoded in the Pubsub topic. For
        AVRO data, this is a schema defined with AVRO schema syntax (https://avro.apache.org/docs/1.10.2/spec.html#schemas).
        For JSON data, this is a schema defined with JSON-schema syntax (https://json-schema.org/).
      nullable: false
      type: str
    subscription:
      description: 'The name of the subscription to consume data. Either a topic or
        subscription must be provided. Format: projects/${PROJECT}/subscriptions/${SUBSCRIPTION}'
      nullable: true
      type: str
    timestampAttribute:
      description: Specifies the name of the attribute that contains the timestamp,
        if any. The timestamp value is expected to be represented in the attribute
        as either (1) a numerical value representing the number of milliseconds since
        the Unix epoch. For example, if using the Joda time classes, Instant.getMillis()
        returns the correct value for this attribute. or (2) a String in RFC 3339
        format. For example, 2015-10-29T23:41:41.123Z. The sub-second component of
        the timestamp is optional, and digits beyond the first three (i.e., time units
        smaller than milliseconds) will be ignored.
      nullable: true
      type: str
    topic:
      description: 'The name of the topic to consume data from. If a topic is specified,  will
        create a new subscription for that topic and start consuming from that point.
        Either a topic or a subscription must be provided. Format: projects/${PROJECT}/topics/${TOPIC}'
      nullable: true
      type: str
  identifier: beam:schematransform:org.apache.beam:pubsub_read:v1
  name: PubsubRead
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/jdbc
  fields:
    autosharding:
      description: ''
      nullable: true
      type: bool
    connectionInitSql:
      description: ''
      nullable: true
      type: Sequence[str]
    connectionProperties:
      description: ''
      nullable: true
      type: str
    driverClassName:
      description: ''
      nullable: false
      type: str
    driverJars:
      description: ''
      nullable: true
      type: str
    jdbcUrl:
      description: ''
      nullable: false
      type: str
    location:
      description: ''
      nullable: true
      type: str
    password:
      description: ''
      nullable: true
      type: str
    username:
      description: ''
      nullable: true
      type: str
    writeStatement:
      description: ''
      nullable: true
      type: str
  identifier: beam:schematransform:org.apache.beam:jdbc_write:v1
  name: WriteToJdbc
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/bigtable
  fields:
    instanceId:
      description: ''
      nullable: false
      type: str
    projectId:
      description: ''
      nullable: false
      type: str
    tableId:
      description: ''
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:bigtable_read:v1
  name: ReadFromBigtable
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/bigquery
  fields:
    query:
      description: The SQL query to be executed to read from the BigQuery table.
      nullable: true
      type: str
    rowRestriction:
      description: Read only rows that match this filter, which must be compatible
        with Google standard SQL. This is not supported when reading via query.
      nullable: true
      type: str
    selectedFields:
      description: 'Read only the specified fields (columns) from a BigQuery table.
        Fields may not be returned in the order specified. If no value is specified,
        then all fields are returned. Example: "col1, col2, col3"'
      nullable: true
      type: Sequence[str]
    tableSpec:
      description: 'The fully-qualified name of the BigQuery table to read from. Format:
        [${PROJECT}:]${DATASET}.${TABLE}'
      nullable: true
      type: str
  identifier: beam:schematransform:org.apache.beam:bigquery_storage_read:v1
  name: StorageReadFromBigQuery
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/pubsublite_write
  fields:
    attributeId:
      description: If set, will set an attribute for each Pubsub Lite message with
        the given name and a unique value. This attribute can then be used in a ReadFromPubSubLite
        PTransform to deduplicate messages.
      nullable: true
      type: str
    attributes:
      description: List of attribute keys whose values will be pulled out as Pubsub
        Lite message attributes.  For example, if the format is `JSON` and attributes
        is `["a", "b"]` then elements of the form `Row(any_field=..., a=..., b=...)`
        will result in Pubsub Lite messages whose payload has the contents of any_field
        and whose attribute will be populated with the values of `a` and `b`.
      nullable: true
      type: Sequence[str]
    errorHandling:
      description: This option specifies whether and where to output unwritable rows.
      nullable: true
      type: Row(output=<class 'str'>)
    format:
      description: 'The encoding format for the data stored in Pubsub Lite. Valid
        options are: RAW,JSON,AVRO'
      nullable: false
      type: str
    location:
      description: The region or zone where the Pubsub Lite reservation resides.
      nullable: false
      type: str
    project:
      description: The GCP project where the Pubsub Lite reservation resides. This
        can be a project number of a project ID.
      nullable: false
      type: str
    topicName:
      description: The name of the topic to publish data into. This will be concatenated
        with the project and location parameters to build a full topic path.
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:pubsublite_write:v1
  name: PubsubliteWrite
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/pubsublite_read
  fields:
    attributeId:
      description: The attribute on incoming Pubsub Lite messages to use as a unique
        record identifier. When specified, the value of this attribute (which can
        be any string that uniquely identifies the record) will be used for deduplication
        of messages. If not provided, we cannot guarantee that no duplicate data will
        be delivered on the Pub/Sub stream. In this case, deduplication of the stream
        will be strictly best effort.
      nullable: true
      type: str
    attributeMap:
      description: Name of a field in which to store the full set of attributes associated
        with this message.  For example, if the format is `RAW` and `attribute_map`
        is set to `"attrs"` then this read will produce elements of the form `Row(payload=...,
        attrs=...)` where `attrs` is a Map type of string to string. If both `attributes`
        and `attribute_map` are set, the overlapping attribute values will be present
        in both the flattened structure and the attribute map.
      nullable: true
      type: str
    attributes:
      description: List of attribute keys whose values will be flattened into the
        output message as additional fields.  For example, if the format is `RAW`
        and attributes is `["a", "b"]` then this read will produce elements of the
        form `Row(payload=..., a=..., b=...)`
      nullable: true
      type: Sequence[str]
    errorHandling:
      description: This option specifies whether and where to output unwritable rows.
      nullable: true
      type: Row(output=<class 'str'>)
    format:
      description: 'The encoding format for the data stored in Pubsub Lite. Valid
        options are: RAW,AVRO,JSON'
      nullable: false
      type: str
    location:
      description: The region or zone where the Pubsub Lite reservation resides.
      nullable: false
      type: str
    project:
      description: The GCP project where the Pubsub Lite reservation resides. This
        can be a project number of a project ID.
      nullable: true
      type: str
    schema:
      description: The schema in which the data is encoded in the Kafka topic. For
        AVRO data, this is a schema defined with AVRO schema syntax (https://avro.apache.org/docs/1.10.2/spec.html#schemas).
        For JSON data, this is a schema defined with JSON-schema syntax (https://json-schema.org/).
      nullable: true
      type: str
    subscriptionName:
      description: The name of the subscription to consume data. This will be concatenated
        with the project and location parameters to build a full subscription path.
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:pubsublite_read:v1
  name: PubsubliteRead
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: 'Writes data to BigQuery using the Storage Write API (https://cloud.google.com/bigquery/docs/write-api).

    This expects a single PCollection of Beam Rows and outputs two dead-letter queues
    (DLQ) that contain failed rows. The first DLQ has tag [FailedRows] and contains
    the failed rows. The second DLQ has tag [FailedRowsWithErrors] and contains failed
    rows and along with their respective errors.'
  destinations:
    python: apache_beam/io/gcp/bigquery
  fields:
    autoSharding:
      description: This option enables using a dynamically determined number of Storage
        Write API streams to write to BigQuery. Only applicable to unbounded data.
      nullable: true
      type: bool
    createDisposition:
      description: 'Optional field that specifies whether the job is allowed to create
        new tables. The following values are supported: CREATE_IF_NEEDED (the job
        may create the table), CREATE_NEVER (the job must fail if the table does not
        exist already).'
      nullable: true
      type: str
    errorHandling:
      description: This option specifies whether and where to output unwritable rows.
      nullable: true
      type: Row(output=<class 'str'>)
    numStreams:
      description: Specifies the number of write streams that the Storage API sink
        will use. This parameter is only applicable when writing unbounded data.
      nullable: true
      type: numpy.int32
    table:
      description: 'The bigquery table to write to. Format: [${PROJECT}:]${DATASET}.${TABLE}'
      nullable: false
      type: str
    triggeringFrequencySeconds:
      description: Determines how often to 'commit' progress into BigQuery. Default
        is every 5 seconds.
      nullable: true
      type: numpy.int64
    useAtLeastOnceSemantics:
      description: This option enables lower latency for insertions to BigQuery but
        may ocassionally duplicate data elements.
      nullable: true
      type: bool
    writeDisposition:
      description: 'Specifies the action that occurs if the destination table already
        exists. The following values are supported: WRITE_TRUNCATE (overwrites the
        table data), WRITE_APPEND (append the data to the table), WRITE_EMPTY (job
        must fail if the table is not empty).'
      nullable: true
      type: str
  identifier: beam:schematransform:org.apache.beam:bigquery_storage_write:v2
  name: StorageWriteToBigQuery
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/bigquery
  fields:
    query:
      description: ''
      nullable: true
      type: str
    queryLocation:
      description: ''
      nullable: true
      type: str
    tableSpec:
      description: ''
      nullable: true
      type: str
    useStandardSql:
      description: ''
      nullable: true
      type: bool
  identifier: beam:schematransform:org.apache.beam:bigquery_export_read:v1
  name: ExportReadFromBigQuery
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/bigtable
  fields:
    instanceId:
      description: ''
      nullable: false
      type: str
    projectId:
      description: ''
      nullable: false
      type: str
    tableId:
      description: ''
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:bigtable_write:v1
  name: WriteToBigtable
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/pubsub_write
  fields:
    attributes:
      description: The set of fields to write as PubSub attributes instead of part
        of the payload.
      nullable: true
      type: Sequence[str]
    attributesMap:
      description: A map field to write as PubSub attributes instead of part of the
        payload.
      nullable: true
      type: str
    errorHandling:
      description: Specifies how to handle errors.
      nullable: true
      type: Row(output=<class 'str'>)
    format:
      description: 'The encoding format for the data stored in Pubsub. Valid options
        are: RAW,AVRO,JSON'
      nullable: false
      type: str
    idAttribute:
      description: If set, will set an attribute for each Cloud Pub/Sub message with
        the given name and a unique value. This attribute can then be used in a ReadFromPubSub
        PTransform to deduplicate messages.
      nullable: true
      type: str
    timestampAttribute:
      description: If set, will set an attribute for each Cloud Pub/Sub message with
        the given name and the message's publish time as the value.
      nullable: true
      type: str
    topic:
      description: 'The name of the topic to write data to. Format: projects/${PROJECT}/topics/${TOPIC}'
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:pubsub_write:v1
  name: PubsubWrite
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/jdbc
  fields:
    connectionInitSql:
      description: ''
      nullable: true
      type: Sequence[str]
    connectionProperties:
      description: ''
      nullable: true
      type: str
    driverClassName:
      description: ''
      nullable: false
      type: str
    driverJars:
      description: ''
      nullable: true
      type: str
    fetchSize:
      description: ''
      nullable: true
      type: numpy.int16
    jdbcUrl:
      description: ''
      nullable: false
      type: str
    location:
      description: ''
      nullable: true
      type: str
    outputParallelization:
      description: ''
      nullable: true
      type: bool
    password:
      description: ''
      nullable: true
      type: str
    readQuery:
      description: ''
      nullable: true
      type: str
    username:
      description: ''
      nullable: true
      type: str
  identifier: beam:schematransform:org.apache.beam:jdbc_read:v1
  name: ReadFromJdbc
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/spanner_write
  fields:
    databaseId:
      description: Specifies the Cloud Spanner database.
      nullable: false
      type: str
    instanceId:
      description: Specifies the Cloud Spanner instance.
      nullable: false
      type: str
    tableId:
      description: Specifies the Cloud Spanner table.
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:spanner_write:v1
  name: SpannerWrite
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/spanner_cdc_read
  fields:
    changeStreamName:
      description: Specifies the change stream name.
      nullable: false
      type: str
    databaseId:
      description: Specifies the Cloud Spanner database.
      nullable: false
      type: str
    endAtTimestamp:
      description: Specifies the end time of the change stream.
      nullable: true
      type: str
    instanceId:
      description: Specifies the Cloud Spanner instance.
      nullable: false
      type: str
    projectId:
      description: Specifies the Cloud Spanner project.
      nullable: false
      type: str
    startAtTimestamp:
      description: Specifies the time that the change stream should be read from.
      nullable: false
      type: str
    table:
      description: Specifies the Cloud Spanner table.
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:spanner_cdc_read:v1
  name: SpannerCdcRead
- default_service: sdks:java:io:google-cloud-platform:expansion-service:shadowJar
  description: ''
  destinations:
    python: apache_beam/io/gcp/csv_write
  fields:
    path:
      description: The file path to write to.
      nullable: false
      type: str
  identifier: beam:schematransform:org.apache.beam:csv_write:v1
  name: CsvWrite

From this config, external transform wrappers are created and stored in appropriate modules. For example, our config gives us the following bigquery_et.py file:

 BigQuery wrappers
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# NOTE: This file contains autogenerated external transform(s)
# and should not be edited by hand.

# pylint:disable=line-too-long

from apache_beam.transforms.external import BeamJarExpansionService
from apache_beam.transforms.external_schematransform_provider import ExternalSchemaTransform

class FileLoadsToBigQuery(ExternalSchemaTransform):

  identifier = "beam:schematransform:org.apache.beam:bigquery_fileloads_write:v1"

  def __init__(
      self,
      create_disposition,
      table_spec,
      write_disposition,
      expansion_service=BeamJarExpansionService(
          "sdks:java:io:google-cloud-platform:expansion-service:shadowJar")):
    """
    :param create_disposition (str):
    :param table_spec (str):
    :param write_disposition (str):
    """
    super().__init__(
        createDisposition=create_disposition,
        tableSpec=table_spec,
        writeDisposition=write_disposition,
        expansion_service=expansion_service)

class StorageReadFromBigQuery(ExternalSchemaTransform):

  identifier = "beam:schematransform:org.apache.beam:bigquery_storage_read:v1"

  def __init__(
      self,
      query=None,
      row_restriction=None,
      selected_fields=None,
      table_spec=None,
      expansion_service=BeamJarExpansionService(
          "sdks:java:io:google-cloud-platform:expansion-service:shadowJar")):
    """
    :param query (str):
      The SQL query to be executed to read from the BigQuery table.
    :param row_restriction (str):
      Read only rows that match this filter, which must be compatible with
      Google standard SQL. This is not supported when reading via query.
    :param selected_fields (Sequence[str]):
      Read only the specified fields (columns) from a BigQuery table. Fields
      may not be returned in the order specified. If no value is specified,
      then all fields are returned. Example: "col1, col2, col3"
    :param table_spec (str):
      The fully-qualified name of the BigQuery table to read from. Format:
      [${PROJECT}:]${DATASET}.${TABLE}
    """
    super().__init__(
        query=query,
        rowRestriction=row_restriction,
        selectedFields=selected_fields,
        tableSpec=table_spec,
        expansion_service=expansion_service)

class StorageWriteToBigQuery(ExternalSchemaTransform):
  """
  Writes data to BigQuery using the Storage Write API
  (https://cloud.google.com/bigquery/docs/write-api).

  This expects a single PCollection of Beam Rows and outputs two dead-letter
  queues (DLQ) that contain failed rows. The first DLQ has tag [FailedRows] and
  contains the failed rows. The second DLQ has tag [FailedRowsWithErrors] and
  contains failed rows and along with their respective errors.
  """
  identifier = "beam:schematransform:org.apache.beam:bigquery_storage_write:v2"

  def __init__(
      self,
      table,
      auto_sharding=None,
      create_disposition=None,
      error_handling=None,
      num_streams=None,
      triggering_frequency_seconds=None,
      use_at_least_once_semantics=None,
      write_disposition=None,
      expansion_service=BeamJarExpansionService(
          "sdks:java:io:google-cloud-platform:expansion-service:shadowJar")):
    """
    :param table (str):
      The bigquery table to write to. Format: [${PROJECT}:]${DATASET}.${TABLE}
    :param auto_sharding (bool):
      This option enables using a dynamically determined number of Storage
      Write API streams to write to BigQuery. Only applicable to unbounded
      data.
    :param create_disposition (str):
      Optional field that specifies whether the job is allowed to create new
      tables. The following values are supported: CREATE_IF_NEEDED (the job
      may create the table), CREATE_NEVER (the job must fail if the table does
      not exist already).
    :param error_handling (Row(output=<class 'str'>)):
      This option specifies whether and where to output unwritable rows.
    :param num_streams (numpy.int32):
      Specifies the number of write streams that the Storage API sink will
      use. This parameter is only applicable when writing unbounded data.
    :param triggering_frequency_seconds (numpy.int64):
      Determines how often to 'commit' progress into BigQuery. Default is
      every 5 seconds.
    :param use_at_least_once_semantics (bool):
      This option enables lower latency for insertions to BigQuery but may
      ocassionally duplicate data elements.
    :param write_disposition (str):
      Specifies the action that occurs if the destination table already
      exists. The following values are supported: WRITE_TRUNCATE (overwrites
      the table data), WRITE_APPEND (append the data to the table),
      WRITE_EMPTY (job must fail if the table is not empty).
    """
    super().__init__(
        table=table,
        autoSharding=auto_sharding,
        createDisposition=create_disposition,
        errorHandling=error_handling,
        numStreams=num_streams,
        triggeringFrequencySeconds=triggering_frequency_seconds,
        useAtLeastOnceSemantics=use_at_least_once_semantics,
        writeDisposition=write_disposition,
        expansion_service=expansion_service)

class ExportReadFromBigQuery(ExternalSchemaTransform):

  identifier = "beam:schematransform:org.apache.beam:bigquery_export_read:v1"

  def __init__(
      self,
      query=None,
      query_location=None,
      table_spec=None,
      use_standard_sql=None,
      expansion_service=BeamJarExpansionService(
          "sdks:java:io:google-cloud-platform:expansion-service:shadowJar")):
    """
    :param query (str):
    :param query_location (str):
    :param table_spec (str):
    :param use_standard_sql (bool):
    """
    super().__init__(
        query=query,
        queryLocation=query_location,
        tableSpec=table_spec,
        useStandardSql=use_standard_sql,
        expansion_service=expansion_service)

Added a new workflow (beam_PostCommit_Python_Xlang_IO_Direct), which is successfully running here

R: @robertwb @tvalentyn @chamikaramj
(manually requesting to make the review bot happy)

You can make the generated transforms more discoverable by importing them in __init.py__ files. that might generating an additional file(s) _external_transforms.py, and importing _external_transforms.py in __init__.py .

So would users be expected to import these transforms from the _et paths directly? One transform per module (which is not how the others are typically arranged, e.g. with the Read and the Write in the same place)? I am also a bit wary of scattering these files all over the place (for both tooling and human reasons).
What if instead they were in a single subdirectory (like with protos) and on import this would then dynamically insert any specifically placed transforms into the right modules (which could include modules that already exist)?

What if instead they were in a single subdirectory (like with protos)

I actually also thought about it and hesitated to suggest only since it felt a bit late in the game to do so.

So would users be expected to import these transforms from the _et paths directly

Having generated files in one directory would also require adding / maintaining one __init__.py file to facilitate the imports.

This seems to be under fairly active work, and doesn't feel quite like a release blocker (new functionality, not a regression, not a bug fix etc).
Is there a compelling reason this should be in the 2.54.0 release instead of getting more time to bake and be in the 2.55.0 release instead? (There's still time, as I have other cherry picks to make tommorrow, and other issues to triage).

Thank you @lostluck, yes best to let it bake and get it in the next release

What if instead they were in a single subdirectory (like with protos)

I like this alternative more actually. I've made some changes to instead generate wrappers inside the apache_beam.transforms._external_transforms package. Inside there, each destination gets its own file (e.g. transforms with destination apache_beam/io will be written to apache_beam/transforms/_external_transforms/io.py).
Then we can make these transforms available to the apache_beam/io package by importing everything from _external_transforms/io.py.
@robertwb @tvalentyn let me know if this is what you had in mind

Having generated files in one directory would also require updating / maintaining only one init.py file to facilitate the imports.

Yeah I'd rather do this manually because I'm wary about altering existing files with a script.

I actually also thought about it and hesitated to suggest only since it felt a bit late in the game to do so.

Not too late! The feedback helps to get this in a good shape before we release it to users

Successful PreCommit run: https://github.com/ahmedabu98/beam/actions/runs/7714357594/job/21026416076

R: @tvalentyn
R: @robertwb
Ready for another review!

External transform config sync check (beam_PreCommit_Xlang_Generated_Transforms) is running green on my fork: https://github.com/ahmedabu98/beam/actions/runs/7832577531/job/21371609985

i think my comments were addressed. LGTM from my side as long as tests pass.

actually, i tried running  gradlew generateExternalTransformWrappers in a clean environment and it failed.
I think the intent of that command should be to regenerate the yaml portion only, but not generate the python wrappers. Wrappers should be generated from the yaml config, when setup.py is called.

you could consider extracting the portion that generates the yaml into a separate script. If you keep the script in one file that can be fine too, but then generating the yaml portion (from gradle) shouldn't require installing jinja.

you could consider extracting the portion that generates the yaml

Makes sense to have the gradle command only generate the config. Then each remote SDK can have its own command to generate wrappers from there (in the future we can sense to combine them all in one command if it makes sense).
We can still have the script in one file, I'll just move imports around to the relevant portions.

Ah, yes, that's right. Hopefully the file generation itself isn't that
slow.
On Mon, Feb 12, 2024 at 3:02 PM Ahmed Abualsaud @.***>
wrote:

@.**** commented on this pull request.
In sdks/python/setup.py
https://github.com/apache/beam/pull/29834#discussion_r1486898981:

 # if exists, this directory will have at least its __init__.py file

 if (not os.path.exists(generated_transforms_dir) or

         len(os.listdir(generated_transforms_dir)) <= 1):

   message = 'External transform wrappers have not been generated '

   if not script_exists:

     message += 'and the generation script `gen_xlang_wrappers.py`'

   if not config_exists:

     message += 'and the standard external transforms config'

   message += ' could not be found'

   warnings.warn(message)

 else:

   warnings.warn(

       'Skipping external transform wrapper generation as they '

       'are already generated.')

 return

out = subprocess.run([

This will make pretty much every run of setup.py slow, right?
Not anymore, since we decided to decouple the generation steps. setup.py
expects the transform config to already exist, so no expansion + discovery
is done here. All it does is generate files based on the existing config.
—
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/29834#discussion_r1486898981, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AADWVAJPRS6PTJHJGTPTHE3YTKNPZAVCNFSM6AAAAABA4ZG2QSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQNZWGUYTSNZSGA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I think we need to put the import back in the try:, except: pass block?

That will work, alternatively, you can have dedicated functions: to generate config, and to generate wrappers, and make necessary imports only in functions where imports are required; you might have to add # pylint: disable=g-import-not-at-top.

@tvalentyn this is already the case, the steps are in different functions. And we're only importing apache_beam when using the function to generate transform configs.
But this function is unusable in a clean setup (one that doesn't have generated modules) unless we make imports like from apache_beam.transforms.xlang.io import * optional.

The newly added precommit that performs a transform config sync check (beam_PreCommit_Xlang_Generated_Transforms) is running green on my fork: https://github.com/ahmedabu98/beam/actions/runs/7933231703/
Will merge after current tests pass

authors: ahmedabu98 damccorm lostluck robertwb tvalentyn


text:
Remove unused it-gcp and it-jdbc modules

Part of #30206
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Besides the reasons in #30206 , for it-gcp itself, it was found in Beam 2.54.0rc1 validation that some changes need to be made in it-gcp component itself: https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1313/files#diff-ca10da4b6f0b454247e31d288a7c6492c0dcb1bb3c78eaab028f2c7716876802. If it-gcp were in Beam repo, it then had to trigger build another rc just for the fix of it-gcp module.

source: gharchive/pull-request

repo: apache/beam

license_type: permissive

license_source: github-api


text:
[BEAM-5428][WIP] State caching in the Python SDK

This is a first draft for a LRU cache and caching of get requests.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@mxm thanks for taking care of this.

Run Portable_Python PreCommit

Run Portable_Python PreCommit

This is now based on the Proto changes in #9440.

@robertwb Could you have a look at the current state of the SDK implementation?

Run JavaPortabilityApi PreCommit

Run Java_Examples_Dataflow PreCommit

Run Python_PVR_Flink PreCommit

Run Java PreCommit

Run Java_Examples_Dataflow PreCommit

Run Python_PVR_Flink PreCommit

Run Python PreCommit

Benchmark with a synthetic source with parallelism of 1 on my local machine:
max_items = 100000
num_keys = 10

class GenerateInput(beam.DoFn):

  def process(self, byte_array):
    key_string = byte_array
    key_int = int(key_string)
    yield (key_int % num_keys, "value" + key_string)

class StatefulProcessing(beam.DoFn):
  count_state_spec = userstate.CombiningValueStateSpec(
    'count', beam.coders.IterableCoder(beam.coders.VarIntCoder()), sum)

  def process(self, kv, count=beam.DoFn.StateParam(count_state_spec)):
    k, v = kv
    count.add(len(v))

(p
 | "Generate data" >> FlinkStreamingImpulseSource().set_message_count(max_items).set_interval_ms(0).with_output_types(bytes)
 | "Reshuffle" >> beam.Reshuffle()
 | "Format Data" >> beam.ParDo(GenerateInput()).with_output_types(KV[int, str])
 | "Stateful Processing" >> beam.ParDo(StatefulProcessing())
)

# elements: 100,000
# bundle size: 10
# number of unique keys: 10
Took 205.37934804  with experiments=state_cache_size=0
Took 129.933783054 with experiments=state_cache_size=10

The version with caching is about 1,59 times faster. These results are consistent across multiple runs. This is of course a bit of an extreme situation in terms of the number of unique keys, but it shows the potential for the caching.

Rebased to master.

Run Python_PVR_Flink PreCommit

Latest changes here: https://github.com/apache/beam/compare/40cb35361234d12bd4ff8f8126aa5128bc07fcba..55588e91ed8e3e25bb661a6202c31e99297e0e79
From my perspective, all comments are addressed now.

Run Portable_Python PreCommit

Py37 somehow broken (also showing up in other builds):
18:34:00 FAILURE: Build completed with 2 failures.
18:34:00
18:34:00 1: Task failed with an exception.
18:34:00 -----------
18:34:00 * What went wrong:
18:34:00 Execution failed for task ':sdks:python:test-suites:tox:py37:testPy37Gcp'.
18:34:00 > Process 'command 'sh'' finished with non-zero exit value 1
18:34:00
18:34:00 * Try:
18:34:00 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
18:34:00 ==============================================================================
18:34:00
18:34:00 2: Task failed with an exception.
18:34:00 -----------
18:34:00 * Where:
18:34:00 Build file '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/dataflow/py37/build.gradle' line: 66
18:34:00
18:34:00 * What went wrong:
18:34:00 Execution failed for task ':sdks:python:test-suites:dataflow:py37:preCommitIT'.
18:34:00 > Process 'command 'sh'' finished with non-zero exit value 1
18:34:00

Run Python PreCommit

Run Portable_Python PreCommit

Python tests broken on master, see https://builds.apache.org/job/beam_PreCommit_Python_Cron/ and https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/

Run Portable_Python PreCommit

Run Python PreCommit

authors: mxm rakeshcusat


text:
grpc 客户端通过 ssl 连接 brpc 服务端

问题描述
如何使用 grpc 客户端通过 ssl 连接 brpc 服务端？
我尝试的方法：
客户端相关代码：
    grpc_impl::ChannelArguments ag;
    ag.SetMaxReceiveMessageSize(DATA_BUF_SIZE);
    ag.SetMaxSendMessageSize(DATA_BUF_SIZE);
    grpc::SslCredentialsOptions sslOptions;
    sslOptions.pem_root_certs = rC; // 对应的文件内容
    sslOptions.pem_private_key = pc;  // 对应的文件内容
    sslOptions.pem_cert_chain = cc; // 对应的文件内容

    auto cha = grpc::CreateCustomChannel(ipAddr, grpc::SslCredentials(sslOptions), ag);
    std::unique_ptr<bimmsg::BimService::Stub> stub_(bimmsg::BimService::NewStub(cha));

    // send msg
    grpc::ClientContext context;
    bimmsg::ProjStandardDataResp reply;
    grpc::Status status = stub_->ProjStandardDataRpc(&context, req, &reply);

服务端相关代码：
    options.mutable_ssl_options()->default_cert.certificate = "/home/xiaogang/CA/server.crt";
    options.mutable_ssl_options()->default_cert.private_key = "/home/xiaogang/CA/server.key";
    options.mutable_ssl_options()->strict_sni = false;
    options.mutable_ssl_options()->verify.ca_file_path = "/home/xiaogang/CA/ca.crt";
    options.mutable_ssl_options()->verify.verify_depth = 1;
    options.mutable_ssl_options()->alpns = "h2";
    options.mutable_ssl_options()->disable_ssl3 = true;

返回错误
grpc 返回的错误码 unavailable（14）, failed to connect all address.

如果不开启ssl，使用普通连接能连接成功吗？

如果不开启ssl，使用普通连接能连接成功吗？
如果不开启ssl，普通连接是可以的

如果不开启ssl，使用普通连接能连接成功吗？

通过抓包分析，能确认的是ssl 握手是成功的，客户端服务端交换了证书

已经解决了，需要在客户端设置对应的SetSslTargetNameOverride

source: gharchive/issue

repo: apache/brpc


text:
.github/common.env: Update Docker images

These images include buildbox-common and buildbox-casd 0.0.58, which should fix #1582.

Tested this locally, works great, thanks !

This fixed the buildgrid test.... but it broke the buildbarn test :(

This fixed the buildgrid test.... but it broke the buildbarn test :(

The buildbarn test succeeded in CI for this branch: https://github.com/apache/buildstream/runs/5492212093?check_suite_focus=true
What kind of failure do you see?

authors: gtristan juergbi

repo: apache/buildstream


text:
Custom Kamelets are not available in Palette

How to reproduce

Create custom Kamelet
Create new route
Try to add Kamelet step with custom Kamelet -> No Kamelet in the Palette

@vidhyasagarj can you take a look?

repo: apache/camel-karavan


text:
Upgrade Zookeeper Testcontainers to 3.8

Description
This PR upgrades Zookeeper Testcontainers to 3.8 for aligning its version with the one used by the camel-zookeeper component.
Target

[x] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[x] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

No corresponding JIRA issue since it's just a minor container version upgrade.
Apache Camel coding standards and style

[x] I checked that each commit in the pull request has a meaningful subject line and body.

[x] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

/component-test camel-zookeeper

authors: orpiske sekikn

repo: apache/camel


text:
[CARBONDATA-3562] Fix for SDK filter queries not working when schema is given explicitly while Add Segment

Problem : Queries will not return correct result from added segment when the schema is given explicitly in case of SDK.
Solution : Handled it by validating based on both column name and column id if it matches for the SDK column.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@ravipesala Please review.

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/717/

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/725/

Build Failed  with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/726/

@manishnalla1994 I think this fix might induce new issues in alter table , like drop column and add back the column with the same name might not work as expected.

@ravipesala when we add the same column then the column id will be different, so it wont match in our case.

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/745/

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/751/

Build Success with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/753/

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/754/

Build Success with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/762/

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/760/

LGTM

LGTM

retest this please

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/784/

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/790/

Build Success with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/792/

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/789/

Build Success with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/797/

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/795/

LGTM

authors: CarbonDataQA ajantha-bhat jackylk manishnalla1994 ravipesala

repo: apache/carbondata


text:
Async chain

This PR relates to https://github.com/apache/cassandra-accord/pull/17

FYI need the following so tests can run
diff --git .build/include-accord.sh .build/include-accord.sh
index 0dc01bf2ba..079c96da1c 100755
--- .build/include-accord.sh
+++ .build/include-accord.sh
@@ -24,8 +24,8 @@ set -o nounset

 bin="$(cd "$(dirname "$0")" > /dev/null; pwd)"

-accord_repo='https://github.com/apache/cassandra-accord.git'
-accord_branch='trunk'
+accord_repo='https://github.com/bdeggleston/cassandra-accord.git'
+accord_branch='async-chain'
 accord_src="$bin/cassandra-accord"

 checkout() {

authors: bdeggleston dcapwell

repo: apache/cassandra


text:
Configure MTU size for KVM

Introduce a new global setting that can be used to customize all MTU values the MTU interface (see libvirt XML - https://libvirt.org/formatnetwork.html#elementsConnect).
First of all we implemented a change for each single nic of a vm. We observed that this is not useful because Linux Bridges / OpenvSwitch with Libvirt always configures bridge network interface MTU
to use the lowest MTU of all ports (devices) on the bridge.
The only useful way to bypass this problem is to set the MTU to the same size for all VMs.
We introduced a global setting.
kvm.mtu.size Set the MTU for Libvirt and KVM (if not set the default MTU will be considered, Attention: If you use OVS the main bridge automatically is adjusted to the guest nic mtu)
if this is not set nothing will happen. after set a value it will be set for all virtual machines.
See https://github.com/apache/cloudstack/pull/3186
ISSUE TYPE

Feature Idea

COMPONENT NAME

KVM, VR

@rohityadavcloud @Pearl1594 will #6426 address this?

This issue has been raised in response of closing an old PR https://github.com/apache/cloudstack/pull/3186 @DaanHoogland - logistics of the why/how of the implementation is not scope of this issue

ok, remark still stands

Closing this, let's revisit this in future.

authors: DaanHoogland rohityadavcloud shwstppr

repo: apache/cloudstack


text:
Live Storage migration Ubuntu 20, does not work

Hello, Team
I tring use Live Migration with Volume Migration with Shared Storage (Ceph) on KVM (Ubuntu 20). But it's failed with:
2022-12-02 09:58:27,651 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-92:ctx-136c4df4 job-442/job-443 ctx-6b9bcc45) (logid:ca6829a0) Failed to migrated vm VM instance {id: "87", name: "i-2-87-VM", uuid: "0780ecb5-a878-4553-8d94-8538a3fff08c", type="User"} along with its volumes. Can't find strategy to move data. Source Host: ceph-osd01, Destination Host: ceph-osd02, Volume UUIDs: c252edd8-5119-4d70-b3b6-65bca49beeba
I'm tring migrate Vm from one Ceph Pool to Another Ceph pool. Both in same Zone and Cluster
CloudStack 4.17.

@happyalexkg is this something you were able to do in older versions? I am not sure if this was ever implemented.

Technically Ceph can do a RBD migration from one to another Ceph storage pool, but this has to be in the same cluster. I don't think CloudStack supports this though.

authors: DaanHoogland happyalexkg wido


text:
Set http level to INFO as default

Description
Avoid logging bytes on direct download on KVM
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

GitHub Issue/PRs
Screenshots (if appropriate):
How Has This Been Tested?

Checklist:

[x] I have read the CONTRIBUTING document.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
Testing
[ ] I have added tests to cover my changes.
[ ] All relevant new and existing integration tests have passed.
[ ] A full integration testsuite with all test that can run on my environment has passed.

No explicit tests needed as changes are purely in log4j file, merging based on code reviews and travis results.

authors: nvazquez rhtyd


text:
vrouter: remove a POSTROUTING rule for port forwarding in VPC router

Description

As discussed in https://github.com/apache/cloudstack/pull/3937#issuecomment-595459008
a rule for port forwarding in VPC router might not be needed.
This fixes the failed result of health check for network VRs.

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[X] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):
How Has This Been Tested?

@DaanHoogland can you kick off test for this PR ?

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1036

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1038

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-1224)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31879 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3952-t1224-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 77 look OK, 6 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
Error
169.23
test_internal_lb.py

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Error
199.77
test_internal_lb.py

test_03_vpc_internallb_haproxy_stats_on_all_interfaces
Error
77.69
test_internal_lb.py

test_04_rvpc_internallb_haproxy_stats_on_all_interfaces
Error
102.11
test_internal_lb.py

test_02_vpc_privategw_static_routes
Error
167.40
test_privategw_acl.py

test_03_vpc_privategw_restart_vpc_cleanup
Error
153.96
test_privategw_acl.py

test_04_rvpc_privategw_static_routes
Error
291.48
test_privategw_acl.py

test_01_single_VPC_iptables_policies
Error
117.03
test_routers_iptables_default_policy.py

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Error
258.67
test_vpc_redundant.py

test_02_redundant_VPC_default_routes
Error
253.00
test_vpc_redundant.py

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Error
167.29
test_vpc_redundant.py

test_04_rvpc_network_garbage_collector_nics
Error
124.70
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Error
208.96
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Error
236.31
test_vpc_redundant.py

test_01_VPC_nics_after_destroy
Error
127.49
test_vpc_router_nics.py

test_02_VPC_default_routes
Error
116.01
test_vpc_router_nics.py

test_01_redundant_vpc_site2site_vpn
Failure
269.21
test_vpc_vpn.py

test_01_vpc_site2site_vpn_multiple_options
Failure
142.69
test_vpc_vpn.py

test_01_vpc_site2site_vpn
Failure
166.50
test_vpc_vpn.py

Trillian test result (tid-1226)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35843 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3952-t1226-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_02_vpc_privategw_static_routes
Failure
215.00
test_privategw_acl.py

test_03_vpc_privategw_restart_vpc_cleanup
Failure
202.71
test_privategw_acl.py

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-1227)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47773 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3952-t1227-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_05_rvpc_multi_tiers
Failure
490.68
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Error
518.61
test_vpc_redundant.py

Testing this one, instead of #3937 - we removed the offending rule, so additional code to handle VPC vs Isolated network VR is not needed.

@weizhouapache testing this one - and after configuring port forward (and stopping iptables on the guest VM) - I can't connect via PF rule to the VM (works fine for VPC though).
Can you please take a look? Otherwise, since we are 1 day from the freeze for 4.14., I'll have to go with #3937 (but I prefer your PR with removed extra lines) ?

@weizhouapache testing this one - and after configuring port forward (and stopping iptables on the guest VM) - I can't connect via PF rule to the VM (works fine for VPC though).
Can you please take a look? Otherwise, since we are 1 day from the freeze for 4.14., I'll have to go with #3937 (but I prefer your PR with removed extra lines) ?

@andrijapanicsb I will look into it today.
one suggestion, can we easily remove a line in systemvm/debian/root/health_checks/iptables_check.py as a workaround ? health check will pass in both network and vpc vr.
https://github.com/apache/cloudstack/pull/3952/files#diff-cf7b342df9a0681cd60e083d7f746bd2

I see it should work @weizhouapache , but let me ping @Pearl1594 for the opinion
thx

We'll do another PR with just that one change, thanks @weizhouapache

@weizhouapache the PR was made, tested (by you and @Pearl1594 - thanks ) and merged - here  https://github.com/apache/cloudstack/pull/3963
Do you want to close this one now or... ?
I will remove the milestone from it - thanks!

@weizhouapache the PR was made, tested (by you and @Pearl1594 - thanks ) and merged - here #3963
Do you want to close this one now or... ?
I will remove the milestone from it - thanks!

considering we have merged #3963 , I think we can move this to 4.14.1
by the way, I have tested the failed test test_vpc_redundant.py in my local testing environment, no issue at all.

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos7 ✖debian. JID-1366

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos7 ✔debian. JID-1509

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-1975)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46271 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3952-t1975-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_deployment_planner.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 78 look OK, 5 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_01_sys_vm_start
Failure
0.06
test_secondary_storage.py

ContextSuite context=TestCpuCapServiceOfferings>:setup
Error
0.00
test_service_offerings.py

test_01_deploy_vm_on_specific_host
Error
35.91
test_vm_deployment_planner.py

test_04_deploy_vm_on_host_override_pod_and_cluster
Error
6.27
test_vm_deployment_planner.py

test_11_migrate_vm
Error
6.30
test_vm_life_cycle.py

test_hostha_enable_ha_when_host_disconected
Error
936.27
test_hostha_kvm.py

test_hostha_enable_ha_when_host_in_maintenance
Error
300.74
test_hostha_kvm.py

@ustcweizhou can you review the test failures?

@ustcweizhou can you review the test failures?

@rhtyd can you re-kick off the tests ?

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos7 ✔debian. JID-1613

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-2223)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33746 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3952-t2223-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

ContextSuite context=Test01DeployVM>:setup
Error
0.00
test_vm_life_cycle.py

ContextSuite context=Test02VMLifeCycle>:setup
Error
0.00
test_vm_life_cycle.py

ContextSuite context=Test03SecuredVmMigration>:setup
Error
0.00
test_vm_life_cycle.py

Test LGTM the failure is due to a regression in Trillian/ansible overiding file (the test_vm_life_cycle.py and has been fixed).

The regression has been fixed, failures not related to this PR.

authors: DaanHoogland andrijapanicsb blueorangutan rhtyd ustcweizhou weizhouapache


text:
[UI] Change permission to access console proxy button

Description
This PR fixes #5623.
Also, some more information that is not present in the issue in question:

Although the proxy console access button does not appear in the UI, users with access permission to the VM in question have access to the console as well. Simply generate the console access link using another user, copy that link, and, on the user without access to the console button, type that link.

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[x] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[x] Trivial

How Has This Been Tested?
Its tested in a local lab, trying to reproduce the steps of the issue #5623.

@blueorangutan ui

@utchoang a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/5625 (SL-JID-798)

Hi @SadiJr Is this PR good to go in 4.16.1 ?

@sureshanaparti I guess yes, but for me, it is fine to go into main only.

@sureshanaparti I guess yes, but for me, it is fine to go into main only.

ok, thanks for confirming @SadiJr

authors: SadiJr blueorangutan sureshanaparti utchoang


text:
unittesting: test resolution is not guaranteed to be exact

a lot of validation pull requests are failing over this test. It should pass but it's failure is environmental in any observed case so far.

Thanks, @DaanHoogland!
LGTM :+1:

LGTM

@DaanHoogland @wilderrodrigues @bhaisaab
What's the purpose of having this unit test?
I've just seen this unit test fail in a travis build (https://travis-ci.org/miguelaferreira/cloudstack/jobs/77300485). What needs to be done in response to this failure?

@miguelaferreira it is proven to be of no use. For what I am concerned we should remove it totally. We can not guarantee any kind of realtime result in/with this test.
The period checked for had been lengthened beyond the 2 ms. If we lengthen the test period over 3 milisecs when testing 2 there is no longer any test being done.

@DaanHoogland: @wilderrodrigues will make a PR to remove it

authors: DaanHoogland bhaisaab miguelaferreira wilderrodrigues


text:
chore: Remove unused function

Which issue does this PR close?

Closes #.
Rationale for this change

What changes are included in this PR?

How are these changes tested?

Thanks @andygrove

repo: apache/datafusion-comet


text:
regression: query error with case when as the group field

Describe the bug
Internal error: Input field name CASE WHEN concat(t.b, Utf8("hello")) = Utf8("test") THEN Utf8("good") ELSE Utf8("bad") END does not match with the projection expression CASE WHEN concat(t.b,Utf8("hello")) = Utf8("test") THEN Utf8("good") ELSE Utf8("bad") END.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

To Reproduce
DataFusion CLI v39.0.0
> create table t(a int, b text);
0 row(s) fetched.
Elapsed 0.019 seconds.

> insert into t values (1,'hello'), (2,'world');
+-------+
| count |
+-------+
| 2     |
+-------+
1 row(s) fetched.
Elapsed 0.019 seconds.

> select (case when concat(b, 'hello') = 'test' THEN 'good' else 'bad' END) as c from t group by c;
Internal error: Input field name CASE WHEN concat(t.b, Utf8("hello")) = Utf8("test") THEN Utf8("good") ELSE Utf8("bad") END does not match with the projection expression CASE WHEN concat(t.b,Utf8("hello")) = Utf8("test") THEN Utf8("good") ELSE Utf8("bad") END.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

Expected behavior
run success
Additional context
I also check in v37 and v38
v37 can run, but v38 report error
DataFusion CLI v37.0.0
❯ create table t(a int, b text);
insert into t values (1,'hello'), (2,'world');
select (case when concat(b, 'hello') = 'test' THEN 'good' else 'bad' END) as c from t group by c;
0 rows in set. Query took 0.006 seconds.

+-------+
| count |
+-------+
| 2     |
+-------+
1 row in set. Query took 0.014 seconds.

+-----+
| c   |
+-----+
| bad |
+-----+
1 row in set. Query took 0.020 seconds.

DataFusion CLI v38.0.0
> create table t(a int, b text);
insert into t values (1,'hello'), (2,'world');
select (case when concat(b, 'hello') = 'test' THEN 'good' else 'bad' END) as c from t group by c;
0 row(s) fetched.
Elapsed 0.008 seconds.

+-------+
| count |
+-------+
| 2     |
+-------+
1 row(s) fetched.
Elapsed 0.007 seconds.

Internal error: Input field name CASE WHEN concat(t.b, Utf8("hello")) = Utf8("test") THEN Utf8("good") ELSE Utf8("bad") END does not match with the projection expression CASE WHEN concat(t.b,Utf8("hello")) = Utf8("test") THEN Utf8("good") ELSE Utf8("bad") END.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

It seems that the projection expression lacks a whitespace between t.b and Utf8("hello").

It might be related to #10088, and could be due to the gap between physical_name and Expr::display_name of case expressions.

It might be related to #10088, and could be due to the gap between physical_name and Expr::display_name of case expressions.

The error originates from the following check between input field names and projection expressions. It seems like failure is the expected behavior. Before v38 the mismatch was being silently ignored. But now they are being detected.
https://github.com/apache/datafusion/blob/b21bf9e8527de82e901c3a61127d63779f230163/datafusion/physical-expr/src/equivalence/projection.rs#L70-L73
The use of any scalar function inside case will fail similarly. An example,
select (case when ends_with(b, 'hello') THEN 'good' else 'bad' END) as c from t group by c;

Internal error: Input field name
CASE WHEN ends_with(t.b, Utf8("hello")) THEN Utf8("good") ELSE Utf8("bad") END
does not match with the projection expression
CASE WHEN ends_with(t.b,Utf8("hello")) THEN Utf8("good") ELSE Utf8("bad") END.

There exists a formatting mismatch in implementations,

The default implementation of display_name does not put a space after the separator like: .join(", ").
https://github.com/apache/datafusion/blob/82f7bf40adb742fc566ce8b280afa344cc85107c/datafusion/expr/src/udf.rs#L339

But the fmt::Display for Expr::ScalarFunction puts a space after the separator:
https://github.com/apache/datafusion/blob/82f7bf40adb742fc566ce8b280afa344cc85107c/datafusion/expr/src/expr.rs#L1904

This is a recent issue which reports the same bug and proposes to fix it by standardizing the separator in display_name. Even though the actual fix is small, many cascading changes are required in existing test snapshots.

A failed query that does not involve separators.
> create table t(a int, b text);
0 row(s) fetched.
Elapsed 0.005 seconds.

> select (case a::bigint when 1 THEN 1 END) as c from t group by c;
Internal error: Input field name CASE CAST(t.a AS Int64) WHEN Int64(1) THEN Int64(1) END does not match with the projection expression CASE t.a WHEN Int64(1) THEN Int64(1) END.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

I think having multiple different types of names for expressions, while also ensuring they are the same, seems somewhat burdensome and error-prone.🤔

I think this Issue is resolved in https://github.com/apache/datafusion/pull/11133 by @jonahgao
However I am not sure if @jonahgao  plans additional changes before we should close this issue -- or if perhaps we should file a new ticket

Let's close it for this case. I'll open a new issue if there are new findings.

authors: alamb doki23 haohuaijin jcsherin jonahgao

repo: apache/datafusion


text:
Fix checkJavaVersion() for 17

This allows Memory to "partially" operate with Java 17.  No off-heap creation of memory allowed.

My mistake, I was editing in a branch derived from the 3.0.0 branch instead of derived from master. My Bad.

repo: apache/datasketches-memory


text:
about openGemini

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
I want to connect to a database called openGemini. How should I write it in the code? This is a Huawei database. I tried importing the package, but the Dolphinscheduler API got an error message,can you give me a idea to called this, I need a example
Documentation Links
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

repo: apache/dolphinscheduler


text:
[Bug-FE][UI Next][V1.0.0-Alpha]The time control displays a default time spacing of 100 years

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
The time control displays a default time spacing of 100 years

What you expected to happen
Interval time is the same day
How to reproduce
The time control displays a default time spacing of 100 years
Anything else
No response
Version
dev
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Consistency with the old UI, turn optimization suggestions


text:
[doc] Add missing doc change by snehal

Add some missing change from conflunce

PTAL @sneh-wha

thanks


text:
[DSIP-39][parameter] Improvement startup parameters/global parameters/project parameters data type

close #15936
Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

Please add the docs。 @sdhzwc

Yesterday I got feedback that the front-end compilation failed, so I'll improve the documentation when I've solved this problem.

Yesterday I got feedback that the front-end compilation failed, so I'll improve the documentation when I've solved this problem.

pnpm run build:prod command failed. You can test it locally.

Codecov Report
Attention: Patch coverage is 55.88235% with 15 lines in your changes are missing coverage. Please review.

Project coverage is 39.93%. Comparing base (d01d3de) to head (df567f3).
Report is 1 commits behind head on dev.

:exclamation: Current head df567f3 differs from pull request most recent head 95b5aa3. Consider uploading reports for the commit 95b5aa3 to get more accurate results

Files
Patch %
Lines

...dolphinscheduler/api/utils/DataTransformUtils.java
42.85%
7 Missing and 1 partial :warning:

...eduler/service/expand/CuringParamsServiceImpl.java
69.23%
3 Missing and 1 partial :warning:

...cheduler/api/service/impl/ExecutorServiceImpl.java
0.00%
1 Missing and 1 partial :warning:

...inscheduler/api/controller/ExecutorController.java
50.00%
1 Missing :warning:

Additional details and impacted files
@@             Coverage Diff              @@
##                dev   #15967      +/-   ##
============================================
+ Coverage     39.85%   39.93%   +0.07%
- Complexity     5060     5082      +22
============================================
  Files          1369     1370       +1
  Lines         45659    45620      -39
  Branches       4872     4846      -26
============================================
+ Hits          18199    18219      +20
+ Misses        25560    25501      -59
  Partials       1900     1900

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

I tried to compile the frontend locally but failed.

I tried to compile the frontend locally but failed.

I tried to compile the frontend locally but failed.

My local is windows environment, using the same method, but reported a bunch of errors, can not lock the problem, but by watching you send the error, I found that onClearSearchTaskType method is not used, I removed, it should be good, or you run a little!

Yesterday I got feedback that the front-end compilation failed, so I'll improve the documentation when I've solved this problem.

pnpm run build:prod command failed. You can test it locally.

done

The database schema sync check failed with pg

I don't see an error related to the code I wrote

The database schema sync check failed with pg

 I don't see an error related to the code I wrote I successfully added the SQL statement.

This is a history problem, not related to this PR.

The database schema sync check failed with pg

 I don't see an error related to the code I wrote I successfully added the SQL statement.

This may due to a history problem

Since the next release will from dev, so I change the milestone to 3.2.2, otherwise this pr cannot be merged before we release 3.2.2

@ruanwenjun I see someone merged in before me, but I can't pull the code from my side, is this a code conflict I need to resolve?

@ruanwenjun That should do it.

Please fix UT. @sdhzwc

authors: SbloodyS codecov-commenter ruanwenjun sdhzwc


text:
DocsUpdate the README.md

Add the new release in Readme.md
Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[x] No Need

Has document been added or modified:

[x] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 37.04 seconds
load time: 488 seconds
storage size: 17180455669 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221110091405_clickbench_pr_43301.html

authors: hello-stephen luzhijing

repo: apache/doris


text:
improvement add more log on rpc error

Proposed changes
Issue Number: close #xxx
Problem summary
A stream load failed, and we found the following log, which missed critical error message.
I guess it might be a rpc error and ErrorText is empty, add more logs.
W1124 16:39:14.329555 874006 tablet_sink.cpp:158] VNodeChannel[2869579-10003], load_id=5c4bf7fcacae1dd8-9a8c82bdc624695, txn_id=2120726589419520, node=172.29.123.127:8060, **, host: 172.29.123.127**
W1124 16:39:15.866672 874004 tablet_sink.cpp:158] VNodeChannel[2869579-10004], load_id=5c4bf7fcacae1dd8-9a8c82bdc624695, txn_id=2120726589419520, node=172.29.123.126:8060, **, host: 172.29.123.127**

Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.31 seconds
load time: 439 seconds
storage size: 17148731817 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221124134504_clickbench_pr_52345.html

authors: hello-stephen zhannngchen


text:
refactor refactor and upgrade dependency tree of FE and support AWS glue catalog

Proposed changes
Issue Number: close #xxx
Problem summary

Spark dpp
Move DppResult and EtlJobConfig to sparkdpp package in fe-common module.
So taht fe-core is longer depends on spark-dpp module, so that the spark-dpp.jar
will not be moved into fe/lib, which reduce the size of FE output.

Modify start_fe.sh
Modify the CLASSPATH to make sure that doris-fe.jar is at front, so that
when loading classes with same qualified name, it will be got from doris-fe.jar firstly.

Upgrade hadoop and hive version
hadoop: 2.10.2 -> 3.3.3
hive: 2.3.7 -> 3.1.3

Override the IHiveMetastoreClient implementations from dependency
ProxyMetaStoreClient.java for Aliyun DLF.
HiveMetaStoreClient.java for origin Apache Hive metastore.
Because I need to modified some of their method to make them compatible with
different version of Hive.

Exclude some unused dependencies to reduce the size of FE output
Now it is only 370MB (Before is 600MB)

Upgrade aws-java-sdk version to 1.12.31

Support AWS Glue Data Catalog

TODO: add doc in next PR
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[x] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.85 seconds
load time: 496 seconds
storage size: 17123016075 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230119071607_clickbench_pr_83924.html

authors: hello-stephen morningman


text:
[refactor](remove unused code) remove many xxxVal structure

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 39.59 seconds
load time: 500 seconds
storage size: 17122195373 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230128014357_clickbench_pr_85306.html

authors: hello-stephen yiguolei


text:
Bug Fix BE crash if scale is invalid

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.64 seconds
stream load tsv:          454 seconds loaded 74807831229 Bytes, about 157 MB/s
stream load json:         39 seconds loaded 2358488459 Bytes, about 57 MB/s
stream load orc:          73 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230314051922_clickbench_pr_113849.html

please add a case.

run buildall

authors: Gabriel39 dataroaring hello-stephen


text:
enhancement print a warning log for long running heartbeat

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fix Change spark dpp default version to 1.2-SNAPSHOT

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fix Update check-pr-if-need-run-build.sh as same as branch master

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fix incorrect result caused by missing output block of union operator(#29677)

Proposed changes
pick #29677 from master into branch-2.0
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
test add testcases and plugin in cloud mode

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

LGTM

authors: gavinchou pingchunzhang


text:
fix fix txn_insert case

Proposed changes
The delete predicate on mow table is implementated in insert, so in txn insert, it is an insert rather than delete operation, the commit order is changed, so the result is changed.

run buildall

run buildall


text:
feature sql block rule support external table

Let SQL Block rule feature support external table:
create sql_block_rule r1 properties("partition_num" = "100", "global" = "true");

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall


text:
The index_parallel task possibly fails for taskId conflict

Affected Version
0.16.x and later.
Description
The index_parallel task possibly fails for taskId conflict of single_phase_sub tasks. Recently I got failures of index_parallel tasks caused by taskId conflict with the following error:
2020-05-29T15:53:24,994 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Exception while running task[AbstractTask{id='index_parallel_ds_name_rollup_for_pv_2020-05-29T07:52:45.317Z', groupId='index_parallel_ds_name_rollup_for_pv_2020-05-29T07:52:45.317Z', taskResource=TaskResource{availabilityGroup='index_parallel_ds_name_rollup_for_pv_2020-05-29T07:52:45.317Z', requiredCapacity=1}, dataSource='ds_name_rollup_for_pv', context={forceTimeChunkLock=true}}]
java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Failed to post task[AbstractTask{id='single_phase_sub_task_ds_name_rollup_for_pv_2020-05-29T07:53:14.337Z', groupId='index_parallel_ds_name_rollup_for_pv_2020-05-29T07:52:45.317Z', taskResource=TaskResource{availabilityGroup='single_phase_sub_task_ds_name_rollup_for_pv_2020-05-29T07:53:14.337Z', requiredCapacity=1}, dataSource='ds_name_rollup_for_pv', context={forceTimeChunkLock=true}}] with error[{"error":"Task[single_phase_sub_task_ds_name_rollup_for_pv_2020-05-29T07:53:14.337Z] already exists!"}].
	at org.apache.druid.client.indexing.HttpIndexingServiceClient.runTask(HttpIndexingServiceClient.java:134) ~[druid-server-0.16.1-incubating.jar:0.16.1-incubating]
	at org.apache.druid.indexing.common.task.batch.parallel.TaskMonitor.submitTask(TaskMonitor.java:268) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
	at org.apache.druid.indexing.common.task.batch.parallel.TaskMonitor.submit(TaskMonitor.java:225) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexPhaseRunner.submitNewTask(ParallelIndexPhaseRunner.java:230) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexPhaseRunner.run(ParallelIndexPhaseRunner.java:138) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runNextPhase(ParallelIndexSupervisorTask.java:268) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSinglePhaseParallel(ParallelIndexSupervisorTask.java:448) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:391) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
	at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:137) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: org.apache.druid.java.util.common.ISE: Failed to post task[AbstractTask{id='single_phase_sub_task_ds_name_rollup_for_pv_2020-05-29T07:53:14.337Z', groupId='index_parallel_ds_name_rollup_for_pv_2020-05-29T07:52:45.317Z', taskResource=TaskResource{availabilityGroup='single_phase_sub_task_ds_name_rollup_for_pv_2020-05-29T07:53:14.337Z', requiredCapacity=1}, dataSource='ds_name_rollup_for_pv', context={forceTimeChunkLock=true}}] with error[{"error":"Task[single_phase_sub_task_ds_name_rollup_for_pv_2020-05-29T07:53:14.337Z] already exists!"}].
	at org.apache.druid.client.indexing.HttpIndexingServiceClient.runTask(HttpIndexingServiceClient.java:119) ~[druid-server-0.16.1-incubating.jar:0.16.1-incubating]
	... 14 more
2020-05-29T15:53:25,003 INFO [task-runner-0-priority-0] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_ds_name_rollup_for_pv_2020-05-29T07:52:45.317Z] status changed to [FAILED].
2020-05-29T15:53:25,010 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_parallel_ds_name_rollup_for_pv_2020-05-29T07:52:45.317Z",
  "status" : "FAILED",
  "duration" : 17567,
  "errorMsg" : "java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Failed to post task[AbstractTask{...",
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}

In my case, there is a daily scheduling script almost simultaneously submits 24 index_parallel tasks(one hour data per task), which aims to compact/create optimisation rollups for a specific datasource. And the maxNumConcurrentSubTasks property is set to 10 for every index_parallel task. Also found that each index_parallel task could submit 10 single_phase_sub tasks in about 1 second and currently we construct taskId for single_phase_sub task by appending a timestamp suffix. I think that explains the taskId conflict.
In local codebase, I enhance uniqueness of taskId of single_phase_sub task by appending a unique hash suffix(calculated from a string of timestamp+randomId) instead of just a timestamp suffix. And it works well.
In my opinion, the index_parallel type of task is the rising star among index type tasks and its parallel processing depends on single_phase_sub tasks, which means it may be very likely to happen that user simultaneously submit more than one index_parallel tasks for a specific datasource. So it's necessary to improve uniqueness of taskId of single_phase_sub` task. I will be glad to hear some feedback from you about this.

I think this was fixed in https://github.com/apache/druid/pull/8612. Could you please try out 0.17 or later versions?

I think this was fixed in #8612. Could you please try out 0.17 or later versions?

Thanks for pointing that out. It should have been fixed.

Thanks. Please feel free to reopen if it still happens.

authors: jihoonson yuanlihan

repo: apache/druid