text:
Can we delete release branches?

@tseaver We have
https://github.com/GoogleCloudPlatform/gcloud-python/tree/release-0.16.1
https://github.com/GoogleCloudPlatform/gcloud-python/tree/releases-0.18.x
and I think they've already served their purpose? If we want an actually pinned commit we can use tags, not branches?

@tseaver Bump

The release-0.16.1 branch can certainly go:  the chance that we will make a 0.16.2 release is infinitessimal.  Until we've made a 0.19.0 release, I would keep the release-0.18.x branch around, on the chance that there is some lingering bug (or another upstream screwup) which might require a 0.18..2 release.

@tseaver, couldn't we just branch off the 0.18.1 tag to make a new release
with a bugfix?
On Tue, Aug 23, 2016 at 9:30 AM, Tres Seaver notifications@github.com
wrote:

The release-0.16.1 branch can certainly go: the chance that we will make
a 0.16.2 release is infinitessimal. Until we've made a 0.19.0 release, I
would keep the release-0.18.x branch around, on the chance that there is
some lingering bug (or another upstream screwup) which might require a
0.18..2 release.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-python/issues/2146#issuecomment-241730635,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALvyLxa2WjvQ2N-XShoOP7Q6hBRXeGcks5qivXrgaJpZM4Jov3r
.

--
Tom Schultz

Just deleted release-0.16.1. @tseaver Bump on the other one

I've deleted the branch, after verifying that it had no drift from 0.18.1.

authors: daspecster dhermes tseaver


text:
Moving Bigtable Row.commit_modifications() into commit().

Fixes #1548.

@jgeewax @jonparrott LMK what you guys think of this

This seems pretty reasonable in terms of the API surface/usage.

:+1:

I think I should update bigtable-data-api.rst as well

Now that I see the change, it feels like there should really be two different row classes in play, rather than having methods which are present, but only available based on the value of append.

Now that I see the change, it feels like there should really be two different row classes in play, rather than having methods which are present, but only available based on the value of append.

I'm somewhat leaning towards this as well. Row seems to have significant variadic behavior based on one argument.

Two classes or three? There are three distinct cases here. Also, do users want three?
The worst / grossest "variadic"-ness to me is in the output of commit().

@tseaver @jonparrott Bump (want to get a release out ASAP so this PR can't block too long).
Two questions:

Two or three row classes? (3 implicit types, "direct", "conditional" and "append")
Would users blanch at having to use 3 separate classes? Could we sufficiently ease this pain with the Table.row() factory?

Also @jgeewax Please weigh in.

Three classes, I think:  the Table.row() factory could figure out which one to instantiate, based on the params passed?

the Table.row() factory could figure out which one to instantiate, based on the params passed?

Agreed.

Working on it now.

This can definitely be dropped since #1557 and others covered it.

source: gharchive/pull-request

authors: dhermes jonparrott tseaver

repo: GoogleCloudPlatform/gcloud-python

license_type: permissive

license_source: bigquery


text:
Link not working

In the. file we have the following:
This notebook focuses on using the Python SDK to call the Vertex AI PaLM API. For more information on using Generative AI Studio without writing code, you can explore [Getting Started with the UI instructions](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/getting-started/getting_started_ui.md)
This link is not working.

I think it should point to the following link:
https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/intro_generative_ai_studio.md

source: gharchive/issue

repo: GoogleCloudPlatform/generative-ai

license_source: github-api


text:
vision: add detect program

This has samples for all 7 vision APIs:

Faces
Labels
Landmarks
Text
Logos
Image Properties (color palette)
Safe Search

Needs:

README
tests
review of crazy code generator

$ go run main.go detect.go gs://python-docs-samples-tests/vision/landmark.jpg
--- detectFaces
No faces found.
--- detectLabels
Labels:
landmark
architecture
arch
ancient history
tourism
monument
ancient roman architecture
facade
temple
place of worship
--- detectLandmarks
Landmarks:
Palace of Fine Arts
--- detectText
Text:
7ぐ

7
ぐ
--- detectLogos
No logos found.
--- detectProperties
Dominant colors:
14.1% - #bbd6e5
4.7% - #624004
15.4% - #e7eef3
1.8% - #885806
5.2% - #c3d2e4
3.0% - #4c3612
1.8% - #7e5816
1.1% - #865e08
2.6% - #d9e5f4
2.5% - #cde8f4
--- detectSafeSearch
Safe Search properties:
Adult: 1
Medical: 1
Spoofed: 1
Violence: 1

@rakyll @jba the code generator and main program are on the side of too clever - what do you think?

repo: GoogleCloudPlatform/golang-samples


text:
feat(spanner): add samples for UpdateDatabase

This PR adds samples and integration tests for UpdateDatabase request.

LGTM apart from a requested change in file name and a general comment: Can we remove the TODOs and keep this PR clean (having only the changes which will be merged ultimately)? You can add a comment on the PR or mention as Note in the title saying that tests are expected to fail because the feature is not enabled in /span/cloud. We can have them locally of course, to test the changes.

@harshachinta: Can we rename the PR title, and description to indicate UpdateDatabase as a feature (not Drop Database Protection)? Also, letting the user know where we are in the program flow would be a good user experience. Can we add some logs for that? Similar to link. You can refer to Node.js PR.

@telpirion Can you please review this PR on high priority?

@tritone
I have made the suggested changes and all the build pass as expected.
Can you please help me with merging the PR if there are no other questions?

authors: harshachinta rajatbhatta


text:
Remove dependency on Abseil

Once the following bugs are fixed we should remove the submodule and remove the dependencies in the makefiles:

[ ] #254
[ ] #253
[ ] #252
[ ] #251
[ ] #250
[ ] #249
[ ] #248

Note to self: remove cctz also.
And remove the packages from the AppVeyor install script.

repo: GoogleCloudPlatform/google-cloud-cpp


text:
API doesn't work on Google App Engine for Business projects

These are projects with IDs of the form "example.com:project-id".

Fixed in Cloud Datastore v1beta3. More information: https://cloudplatform.googleblog.com/2016/04/Google-Cloud-Datastore-gets-faster-cross-platform-API.html

authors: eddavisson pcostell


text:
Test for propagating an HTTP header across multiple requests.

Addresses #836
And also, once we have been able to propagate gRPC OpenCensus headers (see #2389) we should be able to write a similar test for that.

I need to think about this a little more, I think this tests the right thing but is a little hard to follow.  We have a server start another server and call that server.
I think it might be easier to follow and be a bit more explicit if you start two test servers independently of each other and have one call the other.
I'm not opposed to the current approach, just an idea.  Thoughts?

@iantalarico What I was trying to mimick here was several services calling each other. The servers are started on the controller constructor, so they are independent from one another, and one calls the other.
Let me try and alternative to spawning test servers in the controller, using some static variables, and see if it's clearer.

I've added a second commit with an alternative implementation where test servers are created outside the services to add clarity, hopefully. @iantalarico @jskeet PTAL

Yes, same here, tbh, I like the first commit better as well.

I just think it's more explicit about what is happening here when as we are calling from two servers.  However, I'm really okay with either approach and will leave it up to you two if you would prefer to stick with commit 1.

I'll be merging the second commit, I like the first one better, but it's not clearer anyways tbh. On the second commit there are way more comments than on the first one, so it might be easier to understand.

Waiting for all green before merging. Thanks!

authors: amanda-tarafa iantalarico

repo: GoogleCloudPlatform/google-cloud-dotnet


text:
bigtable: Get an individual Instance using InstanceAdminClient

When I want to get the InstanceInfo of a single Instance by name, I need to get a list of Instances using the Instances() method of my client. I then need to loop through the list, comparing the name of each instance to find the instance I want to read. This looks something like:
instances, err := instanceAdminClient.Instances(ctx)
if err != nil {
	return fmt.Errorf("Error retrieving instances. %s", err)
}

var instance *bigtable.InstanceInfo
name := "instance-name"
found := false
for _, i := range instances {
	if i.Name == name {
		instanceInfo = i
		found = true
		break
	}
}
if !found {
	return fmt.Errorf("Error retrieving instance. Could not find %s.", name)
}

I'd like to be able to perform something like this instead:
name := "instance-name"
instance, err := instanceAdminClient.GetInstance(ctx, name)
if err != nil {
	return fmt.Errorf("Error retrieving instance %s. %s", name, err)
}

The method is named InstanceInfo and should do what you want.

Thanks @garye!

authors: garye rileykarson

repo: GoogleCloudPlatform/google-cloud-go


text:
Throw more granular exception on ImageAnnotatorClient.close()

Currently, the image annotator client can throw the generic "Exception" on close. I found this while making the update to close the image annotator client in the vision sample.

I think this may be unavoidable, because the AutoCloseable interface throws Exception: https://docs.oracle.com/javase/7/docs/api/java/lang/AutoCloseable.html#close()

Relevant code in image annotator client: https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/google-cloud-vision/src/main/java/com/google/cloud/vision/v1/ImageAnnotatorClient.java#L232

I see, it sounds like they "strongly encourage implementations to throw more specific exceptions" but it's not clear how they expect you to implement this.

Yes, it isn't clear what we could do. One possibility would be to use a list of Closeable instead of Autocloseable. But I would be worried that at some future point, we would want to add an Autocloseable and need to change it back.
I am closing this for now, but if you want to discuss possibly switching to Closeable or some other option, please reopen!

authors: gguuss michaelbausor

repo: GoogleCloudPlatform/google-cloud-java


text:
BiQuery query method results in java.lang.NoSuchMethodError

I am creating a simple Query against a big query table, however I cannot seem to actually send the query due to an issue with QueryJobConfiguration throwing a java.lang.NoSuchMethodError when it tries to use getQueryParameters(). In fact, looking at the downloaded sources, there seems to be missing dependencies, with my IDE not being able to find com.google.api.services.bigquery.model.QueryParameter in general
I am using the following version of gcloud SDK for java
<dependency>
     <groupId>com.google.cloud</groupId>
     <artifactId>google-cloud-bigquery</artifactId>
     <version>0.22.0-beta</version>
</dependency>

The Truncated error
Caused by: java.lang.NoSuchMethodError: com.google.api.services.bigquery.model.JobConfigurationQuery.getQueryParameters()Ljava/util/List;
	at com.google.cloud.bigquery.QueryJobConfiguration$Builder.<init>(QueryJobConfiguration.java:135)
	at com.google.cloud.bigquery.QueryJobConfiguration$Builder.<init>(QueryJobConfiguration.java:85)
	at com.google.cloud.bigquery.QueryJobConfiguration.fromPb(QueryJobConfiguration.java:813)
	at com.google.cloud.bigquery.JobConfiguration.fromPb(JobConfiguration.java:140)
	at com.google.cloud.bigquery.JobInfo$BuilderImpl.<init>(JobInfo.java:184)
	at com.google.cloud.bigquery.Job.fromPb(Job.java:307)
	at com.google.cloud.bigquery.BigQueryImpl.query(BigQueryImpl.java:595)
	at com.google.cloud.bigquery.BigQueryImpl.query(BigQueryImpl.java:567)

Hi @patmagee, can you provide a bit more info about how to reproduce the issue? When I create a new project using maven and depend only on the google-cloud-bigquery:0.22.0-beta package I am not seeing a NoSuchMethodError.
Thanks!

@michaelbausor this only seems to be an issue when I am NOT using a Legacy SQL query (Ideally i was hoping to used parameterized queries). It appears I can use legacy queries. Right now I am using the QueryRequestBuilder to create a new request (without the parameters). I am doing something similar to the following:
String myQuery = "From .......";
QueryRequest qr = QueryRequest.newBuilder(myQuery).setUseLegacySql(false).build();
QueryResponse response = bigQuery.query(qr);
while (!response.jobCompleted()) {
     Thread.sleep(1000L);
     response = bigQuery.getQueryResults(response.getJobId());
}
return response.getResult();

This was a dependency issue. Someone was using an old version of the API elsewhere that was not documented

authors: michaelbausor patmagee


text:
Refactor PubSub encoding and messages

This change improves PubSub by standardizing how message payload encoding is handled for Rest and gRPC transports, and by easing the handling and acknowledgement of incoming messages.
It introduces some breaking changes to the library.

Added Message to represent incoming messages.
Added PubSubClient::consume() method. This method can be used by pushDelivery servers to convert an HTTP request into an instance of Message.
Subscription::pull() now returns Generator<Message>.
Subscription::acknowledge() and Subscription::modifyAckDeadline() now accept an instance of Message as their first argument.
Subscription::acknowledgeBatch() and Subscription::modifyAckDeadlineBatch() now accept Message[] as their first argument.
Whether message payloads should be encoded and decoded is now determined in the constructor of PubSubClient. This is currently set to true. When gRPC transport is implemented, this value should be determined based on which transport is used.
Incoming message payloads are now decoded internally if necessary. This eliminates the need for users to determine whether to decode message payloads.
The constructor signatures for Topic and Subscription have changed. I fixed the argument order to be more consistent with everything else, and I added an $encode argument. I had previously removed documentation instructing users on directly instantiating either of those classes, as it kind of goes against the spirit of how the library should be used.

I also did a bit of cleanup in the documentation to make the examples more inline with those in other services, and fixed a couple mistakes I found along the way.

This looks like a nice direction to go! I will set the time aside for a full review soon.

As a quick side note, would we be able to get the examples in the readme updated as well as add the message class to our docs?

@tmatsuo @bshaffer this change will affect the pubsub examples you guys have already written, unfortunately. Would you like me to help get those fixed? I'm happy to work with you on making sure the changes needed are highlighted also. Just let me know what would be best for you.

After it's merged and you get a new tag, we'll create PRs on our repos!

LGTM besides otherwise commented.

@dwsupplee all addressed.

authors: dwsupplee jdpedrie tmatsuo

repo: GoogleCloudPlatform/google-cloud-php


text:
Only CSV imports may specify a field delimiter error when attempting to load AVRO files

Trying to load Avro files into a BigQuery table with this command:
Start-BqJob -Destination $dest_table -WriteMode "WriteTruncate" -Load -Type "AVRO" -SourceUris $storageUris
Get the following error:
Start-BqJob : Google.Apis.Requests.RequestError
Only CSV imports may specify a field delimiter. [400]
Errors [
    Message[Only CSV imports may specify a field delimiter.] Location[ - ] Reason[invalid] Domain[global]
]
At line:1 char:8
+ $job = Start-BqJob -Destination $dest_table -WriteMode WriteTruncate  ...
+        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Start-BqJob], GoogleApiException
    + FullyQualifiedErrorId : Google.GoogleApiException,Google.PowerShell.BigQuery.StartBqJob

@trbates the fix is now available in version 1.0.1.4 of the module.

authors: quoctruong trbates


text:
test: add retries to container-registry tests

Fixes #7715

The polling which starts on line 275 failed. Maybe we should increase that or reevaluate the backoff before running multiple times.

I added one more attempt and changed the logic to an exponential backoff method. I didn't want to add many more attempts, given that the sleep will start getting really long. Please LMK if you have any other suggestions.

repo: GoogleCloudPlatform/java-docs-samples


text:
Bug: reflector.go stores objects in cache by name only, broken in namespaces

This is broken for resources that are namespaced.
We need to store objects in the cache by namespace+name.
Example:
Two resources with same name in different namespaces will cause a cache clash.

This is now a major issue in OpenShift where we use Reflectors with ListWatcher implementations that span namespaces (e.g. watch all builds or deployments in order to advance states). Any name collisions on these resources will result in all sorts of wackiness in FIFOs (due to event compression) and caches (due to last-in-wins overwriting).
Is anybody working on the UID fix? If not, we can take a stab at a patch.

@ironcladlou - all things have UIDs now, please feel free to submit a fix.

Do we need a version of the reflector that caches by namespace+name?  If so, we may want to extract "uniqueness" into its own function and make that be something the reflector can be parameterized with.
----- Original Message -----

@ironcladlou - all things have UIDs now, please feel free to submit a fix.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/issues/2294#issuecomment-71219037

I'm not sure the UID fixes the problem for us.  cache.Store has a .Get(name) method.  Some references are based on name and namespace (anything with this name regardless of what its UID is), so we need a logical .Get(namespace, name), .Get(UID) won't do what we want to express.

@deads2k @smarterclayton I have been toying with the idea of separating the key used by the reflector with the interface that a client of a store has.  The reflector needs only a single globally unique value to address objects by (ex: uid), while the clients of the store the reflector is writing into may need other interfaces (ex: namespace/name).  I haven't thought through all the implications of differentiating the interfaces, just recording here before I forget.

@deads2k

I'm not sure the UID fixes the problem for us. cache.Store has a .Get(name) method. Some references are based on name and namespace (anything with this name regardless of what its UID is), so we need a logical .Get(namespace, name), .Get(UID) won't do what we want to express.

Good point. UID doesn't solve it for the client interface.

You don't have to require the reflector to use uid - you just need the person who inits the reflector and the person who calls the reflector agree on what unique is.  I can imagine reflectors that would unique on attributes on the object that is not UID or name+namespace
----- Original Message -----

@deads2k @smarterclayton I have been toying with the idea of separating the
key used by the reflector with the interface that a client of a store has.
The reflector needs only a single globally unique value to address objects
by (ex: uid), while the clients of the store the reflector is writing into
may need other interfaces (ex: namespace/name).  I haven't thought through
all the implications of differentiating the interfaces, just recording here
before I forget.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/issues/2294#issuecomment-71223373

What about on the other side- cache.Store? As a consumer of that interface, I have no knowledge of what namespacing mechanism the reflector might be using to populate the store. All I have is a Get(id) function, an object, and no idea which field to pass to Get or how to make an id from my object that is compatible with whatever scheme the reflector used. I could be misunderstanding the suggested approach.

@ironcladlou Agree, the consumer and reflector might have different ideas about keying. That makes me think there should be distinct interfaces that capture reflector / consumer needs.

Can a reflector be given a cache func?

Similar to what we do for generic.EtcdRegistry?

The consumer of a reflector should be the person who initialized it.  They know what the uniqueness criteria is.

On Jan 23, 2015, at 3:08 PM, Paul Morie notifications@github.com wrote:
@ironcladlou Agree, the consumer and reflector might have different ideas about keying. That makes me think there should be distinct interfaces that capture reflector / consumer needs.
—
Reply to this email directly or view it on GitHub.

That transformation was what I was thinking of

On Jan 23, 2015, at 3:33 PM, Derek Carr notifications@github.com wrote:
Similar to what we do for generic.EtcdRegistry?
—
Reply to this email directly or view it on GitHub.

Still confused- the reflectors aren't consumed. cache.Stores are consumed. The reflector is an orthogonal/external actor which populates a cache.Store.

Then both of them need to agree what constitutes a unique identifier to Get.
----- Original Message -----

Still confused- the reflectors aren't consumed. cache.Stores are consumed.
The reflector is an orthogonal/external actor which populates a cache.Store.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/issues/2294#issuecomment-71266888

For instance, what if you are trying to reflect contents from two deployments at once?
----- Original Message -----

Then both of them need to agree what constitutes a unique identifier to Get.
----- Original Message -----

Still confused- the reflectors aren't consumed. cache.Stores are consumed.
The reflector is an orthogonal/external actor which populates a
cache.Store.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/issues/2294#issuecomment-71266888

@smarterclayton

Then both of them need to agree what constitutes a unique identifier to Get.

No argument there, and I checked out the generic.Etcd implementation- what this does imply though is a change to the cache.Store methods to accept not just an id, but either a runtime.Object or an id/namespace pair. Agree?

An id/namespace pair is somewhat opinionated - it means the cache store only supports 2 means of input.  If your "keyFunc" takes and returns an interface{} then you can pass whatever you want to the store.
----- Original Message -----

@smarterclayton

Then both of them need to agree what constitutes a unique identifier to Get.

No argument there, and I checked out the generic.Etcd implementation- what
this does imply though is a change to the cache.Store methods to accept
not just an id, but either a runtime.Object or an id/namespace pair. Agree?

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/issues/2294#issuecomment-71474163

@smarterclayton

An id/namespace pair is somewhat opinionated - it means the cache store only supports 2 means of input.  If your "keyFunc" takes and returns an interface{} then you can pass whatever you want to the store.

Agree. I think we're on the same page now.

@smarterclayton @ironcladlou also agree, key function seems like the best way.  Thinking through the interface changes:
type Store interface {
    Add(obj interface{})
    Update(obj interface{})
    Delete(obj interface{})
    List() []interface{}
    Get(obj interface{}) (item interface{}, exists bool)

    Replace(objs []interface)

    ContainedIds() util.StringSet
}

One thing I'm not sure about is whether there should be a distinct ContainedIds call now that there's not an idea of id for an object in the rest of the interface.  It implies the existence of a key function.
Thoughts?

@pmorie
That's the interface I have been prototyping. As far as ContainedIds, I don't see the purpose anymore. In the contexts where ContainedIds is used, the IDs are fed back to Get/Update/Delete calls, etc. You could replace those with List, feeding the resulting interface{} instances back to Get/Update/Delete, etc.
ContainedIds in cache.Store is already implemented with a linear iteration over the items map, so it wouldn't even be a performance change in that regard to move the iteration to the caller.

@pmorie One implication I haven't yet seen addressed: if most calls to Store now implicitly execute a pluggable key function, and the key function can fail, any Store function which does keying should probably return an error. Not pretty, but I'm not sure it's safe to omit the error?

Maybe I spoke too soon. I'm still puzzling through how to reproduce this algorithm without exposing the key function from the store (for example).

Maybe I spoke too soon. I'm still puzzling through how to reproduce this algorithm without exposing the key function from the store (for example).

Now I'm not sure why Poller.sync is doing individual Updates with a final Delete of all not-Updated items. Could it be implemented instead by collecting all the new items in a local []interface{} and calling Store.Replace atomically? That sidesteps the requirement to have a means to perform interface{} equality checks via KeyFunc.

I've opened a PR which a POC implementation of namespacing cache support:
https://github.com/GoogleCloudPlatform/kubernetes/pull/3810
Discussion welcome. I'll hold off continuing to work on it until I get some feedback.

I think Poller can be killed.
The only use is in  plugin/pkg/scheduler/factory/factory.go
and there is a TODO there to switch to NewReflector, which I think can now be done.

Even better @erictune

This is serious because it prevents kubelets from seeing two pods with the same name on GCE.

I just wrote a test which fails and shouldn't.

@ironcladlou @pmorie @deads2k is one of you actively working on a fix?

We use poller downstream.  We need status on nodes and pods to be watchable before we can remove it.

On Jan 26, 2015, at 6:18 PM, Eric Tune notifications@github.com wrote:
I think Poller can be killed.
The only use is in plugin/pkg/scheduler/factory/factory.go
and there is a TODO there to switch to NewReflector, which I think can now be done.
—
Reply to this email directly or view it on GitHub.

@erictune

@ironcladlou @pmorie @deads2k is one of you actively working on a fix?

You figured this out already, but for the sake of visibility, let it be known that I am working the issue via #3810.
P.S., we are currently using Poller downstream because of our inability to receive watch events in response to Pod container state changes (which aren't persisted in etcd). We'd really love to ditch Poller, but can't without the watch functionality.

This issue should be closed by https://github.com/GoogleCloudPlatform/kubernetes/pull/3810

#3810 is merged.

authors: deads2k derekwaynecarr erictune goltermann ironcladlou pmorie smarterclayton

repo: GoogleCloudPlatform/kubernetes


text:
Is there any meaning to a service with an empty selector?

For example, kubernetes-ro service appears with empty selector (this service was created by the hack/local cluster)  - is that ok? What meaning such service has if it has no entity to fwd traffic to?

kubernetes and kubernetes-ro are currently "virtual" - they are not backed
by running pods, but by the master itself.
On Thu, Jan 22, 2015 at 5:27 AM, abonas notifications@github.com wrote:

For example, kubernetes-ro service appears with empty selector (this
service was created by the hack/local cluster) - is that ok? What meaning
such service has if it has no entity to fwd traffic to?
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712.

@thockin, if that's internal impl, why is it exposed to user?
let's say a client pulls all entities and builds a UI around it.
then a service with no selector arrives, and it's internal to kube - how the client should know not to present it/touch it? (similar to the "net" container in pods)

It's exposed exactly FOR users - so they can find the apiserver.  It has a
corresponding Endpoints object, it gets a portal IP and a kube-proxy rule,
and it gets service env vars and a DNS name.
The fact that it is virtual is an implementation detail.  That it exists is
all users should need to know.
On Thu, Jan 22, 2015 at 12:43 PM, abonas notifications@github.com wrote:

@thockin https://github.com/thockin, if that's internal impl, why is it
exposed to user?
let's say a client pulls all entities and builds a UI around it.
then a service with no selector arrives, and it's internal to kube - how
the client should know not to present it/touch it? (similar to the "net"
container in pods)
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71094857
.

And FWIW, the master components may one day actually be pods...
On Thu, Jan 22, 2015 at 12:47 PM, Tim Hockin thockin@google.com wrote:

It's exposed exactly FOR users - so they can find the apiserver.  It has a
corresponding Endpoints object, it gets a portal IP and a kube-proxy rule,
and it gets service env vars and a DNS name.
The fact that it is virtual is an implementation detail.  That it exists
is all users should need to know.
On Thu, Jan 22, 2015 at 12:43 PM, abonas notifications@github.com wrote:

@thockin https://github.com/thockin, if that's internal impl, why is
it exposed to user?
let's say a client pulls all entities and builds a UI around it.
then a service with no selector arrives, and it's internal to kube - how
the client should know not to present it/touch it? (similar to the "net"
container in pods)
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71094857
.

@thockin thanks for the explanation. so any general rule how to distinguish kube pods/services from user defined?

Do we need to distinguish them?  We could add labels, I guess, if it is really important to distinguish them.
@bgrant0607

sometimes users would want to see only the things they defined and not implementation details of the system.

One could argue this is why the users should be using labels.  I'm not
against defining labels for "special" stuff, but it's new territory, so we
should enter with caution.
On Thu, Jan 22, 2015 at 1:02 PM, abonas notifications@github.com wrote:

sometimes users would want to see only the things they defined and not
implementation details of the system.
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71097993
.

I think the right answer might be to put them in a special "cluster" or
"system" namespace.
On Thu, Jan 22, 2015 at 1:04 PM, Tim Hockin notifications@github.com
wrote:

One could argue this is why the users should be using labels. I'm not
against defining labels for "special" stuff, but it's new territory, so we
should enter with caution.
On Thu, Jan 22, 2015 at 1:02 PM, abonas notifications@github.com wrote:

sometimes users would want to see only the things they defined and not
implementation details of the system.
--
Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71097993

.

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71098302
.

See #3293
If we put them in a "kubernetes" ns, the DNS name would be
kubernetes.kubernetes.kubernetes.local :)
I'm not against segmenting them into a different namespace.  We just need
to decide on what that namespace is called, plumb through that
configuration, decide on compatibility for a short while or not, and decide
if that namespace is "special" or just "different" - i.e. should the
skydns, heapster, etc pods go into this namespace?
On Thu, Jan 22, 2015 at 1:12 PM, Brendan Burns notifications@github.com
wrote:

I think the right answer might be to put them in a special "cluster" or
"system" namespace.
On Thu, Jan 22, 2015 at 1:04 PM, Tim Hockin notifications@github.com
wrote:

One could argue this is why the users should be using labels. I'm not
against defining labels for "special" stuff, but it's new territory, so
we
should enter with caution.
On Thu, Jan 22, 2015 at 1:02 PM, abonas notifications@github.com
wrote:

sometimes users would want to see only the things they defined and not
implementation details of the system.
--
Reply to this email directly or view it on GitHub
<

https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71097993

.

--
Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71098302>

.

--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71099565
.

@abonas As @thockin said, the services are there for applications that need to contact the apiserver. Additionally, nil selectors can be used by users to import external services.
Adding labels to the kubernetes services won't help. Users need to manage their objects using labels.
/cc @smarterclayton

Yeah, lots of consumers will expose "special" services.  IN fact, in many clusters the "special" services will outweigh the unspecial ones.  At this point, the namespace they are in is completely arbitrary, and I think because it is a name, it should be free for the end user to choose where they are surfaced.

Per the proposal here: #3613, I think the default namespace should continue to hold these services, and they require no special labels.  I sense @abonas comment is made without thinking through what Kubernetes operators do to bootstrap a cluster.  I think it's reasonable that default namespace holds everything you need to run, and operators may or may not choose to segment their data in other namespaces.  If they want to not see this information, its an exact reason for why they should use their own namespace.  I don't think we need anything new here.

@smarterclayton - I'm not clear on your meaning.  Did you mean that we
should make the "cluster admin" namespace a config param that everyone
knows about and can be different cluster-to-cluster?  Or did you mean that
default is as arbitrary a name as any other?
I'd be fine moving things out of default into "kube" or "kubernetes" or
"cluster" or something, but I am less convinced that a flag is a good thing.
On Thu, Jan 22, 2015 at 1:45 PM, Clayton Coleman notifications@github.com
wrote:

Yeah, lots of consumers will expose "special" services. IN fact, in many
clusters the "special" services will outweigh the unspecial ones. At this
point, the namespace they are in is completely arbitrary, and I think
because it is a name, it should be free for the end user to choose where
they are surfaced.
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71105212
.

@thockin - wherever we stick this stuff, it should be the namespace that is initially bootstrapped, which today is default.  I continue to believe that is as good a name as any other.  I do not see a need to do more here.

I think as soon as you bring naming into the picture, making decisions based on names gets very tricky (what if I want my logging service to be everywhere?).  If there is a magic namespace that exposes things to the rest of the cluster, we should ask whether there's only one magic namespace, or multiple?  Are services supposed to cross namespaces?  It's great that pods can find the master... but that's not the only thing they'll need to find in each namespace.  And baking in special names (or customizable names) into your app code is fragile, because then your code only works inside Kubernetes.
I'm just trying to pump the brakes on special services prior to us designing them... :)
----- Original Message -----

@smarterclayton - I'm not clear on your meaning.  Did you mean that we
should make the "cluster admin" namespace a config param that everyone
knows about and can be different cluster-to-cluster?  Or did you mean that
default is as arbitrary a name as any other?
I'd be fine moving things out of default into "kube" or "kubernetes" or
"cluster" or something, but I am less convinced that a flag is a good thing.
On Thu, Jan 22, 2015 at 1:45 PM, Clayton Coleman notifications@github.com
wrote:

Yeah, lots of consumers will expose "special" services. IN fact, in many
clusters the "special" services will outweigh the unspecial ones. At this
point, the namespace they are in is completely arbitrary, and I think
because it is a name, it should be free for the end user to choose where
they are surfaced.
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71105212
.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71105921

oh, to be clear, I don't want "special" services - I am merely asking if
moving non-user-defined objects (where that is defined very loosely) out of
"default" is worthwhile.  I tend to agree with Derek that it's not worth
energy.
On Thu, Jan 22, 2015 at 1:54 PM, Clayton Coleman notifications@github.com
wrote:

I think as soon as you bring naming into the picture, making decisions
based on names gets very tricky (what if I want my logging service to be
everywhere?). If there is a magic namespace that exposes things to the rest
of the cluster, we should ask whether there's only one magic namespace, or
multiple? Are services supposed to cross namespaces? It's great that pods
can find the master... but that's not the only thing they'll need to find
in each namespace. And baking in special names (or customizable names) into
your app code is fragile, because then your code only works inside
Kubernetes.
I'm just trying to pump the brakes on special services prior to us
designing them... :)
----- Original Message -----

@smarterclayton - I'm not clear on your meaning. Did you mean that we
should make the "cluster admin" namespace a config param that everyone
knows about and can be different cluster-to-cluster? Or did you mean that
default is as arbitrary a name as any other?
I'd be fine moving things out of default into "kube" or "kubernetes" or
"cluster" or something, but I am less convinced that a flag is a good
thing.
On Thu, Jan 22, 2015 at 1:45 PM, Clayton Coleman <
notifications@github.com>
wrote:

Yeah, lots of consumers will expose "special" services. IN fact, in
many
clusters the "special" services will outweigh the unspecial ones. At
this
point, the namespace they are in is completely arbitrary, and I think
because it is a name, it should be free for the end user to choose
where
they are surfaced.
--
Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71105212

.

Reply to this email directly or view it on GitHub:

https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71105921
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71106800
.

@smarterclayton agree with you that lots of consumers will expose "special" services
We should not use the (namespace + k8s-service + skydns) mechanism for publishing the location of those special services.   We should decouple the choice of namespace to run the pods of the service in from the well-known dns address of special services.
Otherwise, it will be hard to move the pods of the service to another namespace, which people will definitely want to do as they grow.
Instead, we need a place to publish well-known services which is not tied to the namespace of the pods of that service.

@etune you open a whole new topic.  I've noodled with the idea of a Name
resource type.  We would generate a Name for each Service as
"service.namespace", but users could also add their own Name resources.
On Thu, Jan 22, 2015 at 2:41 PM, Eric Tune notifications@github.com wrote:

@smarterclayton https://github.com/smarterclayton agree with you that lots
of consumers will expose "special" services
We should not use the (namespace + k8s-service + skydns) mechanism for
publishing the location of those special services. We should decouple the
choice of namespace to run the pods of the service in from the well-known
dns address of special services.
Otherwise, it will be hard to move the pods of the service to another
namespace, which people will definitely want to do as they grow.
Instead, we need a place to publish well-known services which is not tied
to the namespace of the pods of that service.
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71114169
.

@bgrant0607  I'd like to understand better your comment regarding nil selectors used to export external services - can you elaborate? give an example? my understanding is that service redirects traffic to pods based on labels. with nil selector service does nothing, so why is it a service in the first place?
@erictune  I'm pro decoupling special services/internal implementation services from user defined entities. whether by separate namespace, or by saying that the default namespace is reserved, or by any other solution. because otherwise internal implementation and user defined content are mixed and that's no good for a proper management.
@simon3z  @itamarh just tagging you so you could see this issue and the related discussions.

Services abstract backends - they don't have to abstract pods.  Some examples:

You want to have an external database cluster in production, but in test you use your own databases.  You would have a service for both and use the environment variables, but the production service has a nil selector with its endpoints set to the database cluster.
You want to have an empty service because you're reserving a name (that isn't accessible yet) but will be in the future
You want to point your service to a service in another namespace or on another cluster.  You set the endpoints for your service to the publicIPs of the other service.

----- Original Message -----

@bgrant0607  I'd like to understand better your comment regarding nil
selectors used to export external services - can you elaborate? give an
example? my understanding is that service redirects traffic to pods based on
labels. with nil selector service does nothing, so why is it a service in
the first place?
@erictune  I'm pro decoupling special services/internal implementation
services from user defined entities. whether by separate namespace, or by
saying that the default namespace is reserved, or by any other solution.
because otherwise internal implementation and user defined content are mixed
and that's no good for a proper management.
@simon3z  @itamarh just tagging you so you could see this issue and the
related discussions.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71190344

@smarterclayton , thanks.
regarding the last point - I was under impression (based on some other discussion) that there would not be cross namespace interaction? that entities can can access only entities in the same namespace?

Cross-namespace is currently allowed and I think has to be allowed, though
perhaps (in the fullness of time) that can be policy based rather than wide
open.
On Fri, Jan 23, 2015 at 12:24 PM, abonas notifications@github.com wrote:

@smarterclayton https://github.com/smarterclayton , thanks.
regarding the last point - I was under impression (based on some other
discussion) that there would not be cross namespace interaction? that
entities can can access only entities in the same namespace?
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71259439
.

@thockin please see this correspondence. my understanding is that there is isolation
https://github.com/GoogleCloudPlatform/kubernetes/issues/3173#issuecomment-68711242
https://github.com/GoogleCloudPlatform/kubernetes/issues/3173#issuecomment-68713577

Responded there.  I think there's some less-than-crystal-clear words in
play.
On Fri, Jan 23, 2015 at 2:55 PM, abonas notifications@github.com wrote:

@thockin https://github.com/thockin please see this correspondence. my
understanding is that there is isolation
#3173 (comment)
https://github.com/GoogleCloudPlatform/kubernetes/issues/3173#issuecomment-68711242
#3173 (comment)
https://github.com/GoogleCloudPlatform/kubernetes/issues/3173#issuecomment-68713577
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71280685
.

+1 to Eric's comment about ensuring there is a consistent manner for system
services like kubernetes-ro, if that's not fixes for all clusters it's
going to be a nightmare to write reusable containers for multiple different
clusters.
I don't care if we do that by using a constant namespace, or by giving
system services two different names, but there needs to be at least one
name that is constant.
Regarding default vs. cluster namespace, I think the issue is one of
clutter as a user is starting to understand k8s, you turn up a new cluster,
launch a pod, or service, and it's hard to find your one object on the sea
of system created pods and services.  It's visual clutter that confuses
people as they onboard.
On Jan 23, 2015 3:01 PM, "Tim Hockin" notifications@github.com wrote:

Responded there. I think there's some less-than-crystal-clear words in
play.
On Fri, Jan 23, 2015 at 2:55 PM, abonas notifications@github.com wrote:

@thockin https://github.com/thockin please see this correspondence. my
understanding is that there is isolation
#3173 (comment)
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/3173#issuecomment-68711242
#3173 (comment)
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/3173#issuecomment-68713577
--
Reply to this email directly or view it on GitHub
<
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71280685
.

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-71281400
.

@abonas - can you drop me a line (my github name at google) - I want to ask
you something, and you have no contact info :)
On Thu, Jan 22, 2015 at 5:27 AM, abonas notifications@github.com wrote:

For example, kubernetes-ro service appears with empty selector (this
service was created by the hack/local cluster) - is that ok? What meaning
such service has if it has no entity to fwd traffic to?
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712.

@brendandburns, sorry for naive question but why would I care as an user that wants to run its existing workload on Kubernetes about kubernetes-ro ? If I write an application that uses kubernetes-ro then I'm locked into Kubernetes.
I understand the need for kubernetes-ro being accessible by cluster level services such as monitoring, logging. I also can imagine that a power user might want to start its own cluster services that use kubernetes-ro. I don't see clearly why would majority of users care about kubernetes-ro being visible from their container.
I see a reason for the cluster services being visible and accessible for Kubernetes developer, but from an user perspective such services only make the system more complex: why do I see these services (pods,rc) ? are they related to my application ? should I care about them (especially when I use hosted Kubernetes) ? why does my container get all these environment variables for cluster services ?

I will update service.md file to explain why one would have service without a selector.

Right now, every service goes everywhere for simplicity.  Name spacing and
filtering probably make sense in the long run, but it's not a high priority
for v1.0
Brendan
On Mar 4, 2015 5:02 AM, "Rafał Sokołowski" notifications@github.com wrote:

I will update service.md file to explain why one would have service
without a selector.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/3712#issuecomment-77153469
.

authors: abonas bgrant0607 brendandburns derekwaynecarr erictune rsokolowski smarterclayton thockin


text:
Revert "Fix scheduler crashes when profiling is enabled"

Reverts GoogleCloudPlatform/kubernetes#6623

Creating rollback to resolve issue #6644.

LGTM, sorry for the breakage. I think I know what to do to fix it.

authors: fabioy vmarmol


text:
Kubelet: record the timestamp correctly in the runtime cache

Currently, the timstamp is recorded after getting the pods from the container
runtime. This could lead to pod workers seeing stale pods which were retrieved
before their last sync finished. This change fixes that.

/cc @bprashanth

Since there are multiple pod workers updating the cache simultaneously, you also need to ensure that the timestamps always move forward to prove that each pod worker sees the effects of its previous update (which is really what's needed).
Also it would be great to get a ut or something, though I realize runtime_cache doesn't have any so upto you.

Please add a test that illustrates the failure you're correcting-- otherwise it will probably break again in the future, as this change is not obvious to the casual reader (me).

/subscribe
/ref #7561

Also it would be great to get a ut or something, though I realize runtime_cache doesn't have any so upto you.

@bprashanth, what do you mean by getting a "ut"?

A unittests, preferably one that fails without your fix (so straddles pod_worker and runtime_cache)

Since there are multiple pod workers updating the cache simultaneously, you also need to ensure that the timestamps always move forward to prove that each pod worker sees the effects of its previous update (which is really what's needed).

I agree that the timestamp should always move forward. In fact, pod workers won't have this problem because they call ForceUpdateIfOlder, which acquires the lock before calling docker. The calls are serialized by the lock and the timestamp always moves forward. It's different for the updating thread though since it grabs the lock after getting the pods from docker (which I assume is a performance optimization).
I updated the PR with the timestamp checking, along with some unit tests.

In fact, pod workers won't have this problem because they call ForceUpdateIfOlder, which acquires the lock before calling docker. The calls are serialized by the lock and the timestamp always moves forward.

you can still have ForceUpdateIfOlder set the cache and timestamp in pod worker1, and before pod worker1 does a GetPods the timestamp and cache get clobbered with an older cache from pod worker 2 if you don't check timestamps.

you can still have ForceUpdateIfOlder set the cache and timestamp in pod worker1, and before pod worker1 does a GetPods the timestamp and cache get clobbered with an older cache from pod worker 2 if you don't check timestamps.

I don't think that'd happen given that pod worker2 will update the pods AFTER pod worker1, so the cache would only have newer pods than pod worker1 expected. Newer pods are okay though.

I suppose you're safe with just handling the background thread case, which could potentially get invoked from any pod worker. The podworkers themselves should be safe as long as they update both timestamp and pods with the lock as you mentioned.

LGTM

authors: bprashanth lavalamp wojtek-t yujuhong


text:
update example/guestbook-go to v1beta3

update example/guestbook-go to v1beta3
@nikhiljindal, could you double check that we don't need to modify _src/main.go? Thanks.

LGTM. Thanks for the change.

authors: caesarxuchao piosz


text:
Catch 404 error when deleting QueuedResource

Description
If the TPU has already been deleted by the TPU cleanup job, we should not return an error status.
Checklist
Before submitting this PR, please make sure (put X in square brackets):

[x] I have performed a self-review of my code.
[x] I have necessary comments in my code, particularly in hard-to-understand areas.
[x] I have made or will make corresponding changes to the doc if needed.

Do you know why the resource deletion request can be lost during the clean up process?

We don't actually monitor the individual delete operations, we monitor the parent QueuedResource to see if it goes into SUSPENDED. (There's actually some lag between when the nodes are all gone and we can delete the QR).
The TPU cleanup task deletes the QR separately sometimes, and then we get a 404 on get_queued_resource

authors: will-cromar zpcore

repo: GoogleCloudPlatform/ml-auto-solutions


text:
copy from go.opentelemetry.io/otel/exporters/trace/stackdriver

copied from go.opentelemetry.io/otel at commit 79de90a31305151698d8136ae2848b1c01371160

LGTM

authors: rghetia ymotongpoo


text:
custom query row validation has inconsistent column case to table validation

This example came up on an Oracle to PostgreSQL validation. Default identifier case in Oracle is uppercase. Default identifier case in PostgreSQL is lowercase.
Oracle test table:
create table dvt_test.tab_case (id number(6), col_data varchar2(20));
insert into dvt_test.tab_case values (1,'Hello');
commit;

PostgreSQL test table:
create table dvt_test.tab_case (id numeric(6), col_data varchar(20));
insert into dvt_test.tab_case values (1,'Hello');

Table level row validation
data-validation validate row \
 -sc ora_local -tc pg_local \
 -tbls=dvt_test.tab_case \
 --primary-keys=id \
 --comparison-fields=col_data

This works fine and I can see from the debugger that the table objects in query_builder.compile() are:
AlchemyTable: tab_case
  id       decimal(6, 0)
  col_data string

AlchemyTable: tab_case
  id       decimal(6, 0)
  col_data string

Notice the upper case Oracle columns have been converted to lower case.
Custom query row validation
data-validation validate custom-query row \
 -sc ora_local -tc pg_local \
 --source-query='select * from dvt_test.tab_case' \
 --target-query="select * from dvt_test.tab_case" \
 --primary-keys=id \
 --comparison-fields=col_data
...
    raise com.IbisTypeError(
ibis.common.exceptions.IbisTypeError: Column 'col_data' is not found in table. Existing columns: 'ID', 'COL_DATA'.

I can see from the debugger that the table object in query_builder.pt.compile() is:
SQLQueryResult
  query: 'select * from dvt_test.tab_case'
  schema:
    ID       decimal(6, 0)
    COL_DATA string

The columns remain upper case.
If I change --comparison-fields to upper case we then fail on the PostgreSQL side.
data-validation validate custom-query row \
 -sc ora_local -tc pg_local \
 --source-query='select * from dvt_test.tab_case' \
 --target-query="select * from dvt_test.tab_case" \
 --primary-keys=id \
 --comparison-fields=COL_DATA
...
    raise com.IbisTypeError(
ibis.common.exceptions.IbisTypeError: Column 'COL_DATA' is not found in table. Existing columns: 'id', 'col_data'.

The workaround is to force the Oracle query to result in a lower case column.
I think we are inconsistent somewhere in the code. For table validations column names are lower cased, for custom query validations they are not.

I found that column names in an Ibis table object are lower cased, in sqla_table in the line below from ibis/backends/base/sql/alchemy/__init__.py
519  	        sqla_table = self._get_sqla_table(name, database=database, schema=schema)

I could not get to where exactly they are being lower cased.
I've made custom query column names also lower case via a change in DVT's _metadata method override of the Oracle client. I'm not particularly happy with it (feels grubby) but I'm not sure where to look for a better solution. Any thoughts @nehanene15 ?

I wonder if this has to do with us building configs? For column/row validations we use casefold() to lowercase column names so the user can provide case insensitive names. Here is an example for column aggs/PKs.
I wonder if the config generated for regular validation and custom query is different since custom query uses the SQLQueryResult instead of the SQLAlchemyTable.
I'm also curious if other DB clients have this issue with custom query?

I can get the same exception if I pass upper case column names to SQL Server:
data-validation -v validate custom-query row -sc sqlserver -tc bq \
 --source-query="select ID,COL_INT8,COL_INT16,COL_INT32 from pso_data_validator.dvt_core_types" \
 --target-query="select id,col_int8,col_int16,col_int32 from pso_data_validator.dvt_core_types" \
 --primary-keys=id --comparison-fields="col_int16,col_int32"
...
  File "/home/user/professional-services-data-validator/env/lib/python3.10/site-packages/ibis/expr/operations/generic.py", line 39, in __init__
    raise com.IbisTypeError(
ibis.common.exceptions.IbisTypeError: Column 'col_int16' is not found in table. Existing columns: 'ID', 'COL_INT8', 'COL_INT16', 'COL_INT32'.

I guess the difference is that Oracle upper cases columns by default and most other systems lower case them. I didn't expect SQL Server to honour the case I used in the source query.
It seems this is a wider problem than just Oracle so, for now, I'll back out my previous change.

Note for self.
When validating an Oracle table, not custom query, we pass through
professional-services-data-validator/env/lib/python3.10/site-packages/sqlalchemy/dialects/oracle/base.py
    @reflection.cache
    def get_columns(self, connection, table_name, schema=None, **kw):
...
        c = connection.execute(sql.text(text), params)

        for row in c:
            colname = self.normalize_name(row[0])
            orig_colname = row[0]

Notice the use of self.normalize_name(row[0]). row[0] matches the data dictionary and is upper case by default in Oracle. self.normalize_name(row[0]) converts it to lower case.
The code to normalize_name() is here: sqlalchemy/engine/default.py(769)normalize_name()
I think we need to build normalize_name() into the flow for custom-query somewhere.


text:
Switch base image from base to debian8

The newest debian images for jessie (gcr.io/google-appengine/debian8:latest) now has all the extra env variables and ca-certs that were in base.
cc @dlorenc

Looks good to me, but I hope Jon can also look at this.

authors: duggelz sharifelgamal

repo: GoogleCloudPlatform/python-runtime


text:
Add XUnit CI enforcement script

As title.
Question
This script currently works on a per-sample-directory basis - should I reconfigure it to work on a per-repo one?

Thanks Ace! Could you please add a readme for this directory with some explanation of what it is and some instructions?

authors: ace-n alixhami


text:
spanner: add delete_data sample

Add the missing spanner_delete_data sample.

@jiren Do you have any idea why the test ruby2.4 failed? It seems not to be related to my changes.

@jiren Do you have any idea why the test ruby2.4 failed? It seems not to be related to my changes.

@hengfengli  It is working as expected for spanner.  It will skip tests for ruby 2.4
Issue is due tobigquery tests are failing. CI logs
 1) Error:
Load table from CSV file on GCS#test_0001_loads a new table from a CSV file on GCS:
Google::Cloud::AlreadyExistsError: duplicate: Already Exists: Dataset cloud-samples-ruby-test-4:test_dataset_1591664708
    /usr/local/bundle/gems/google-cloud-bigquery-1.21.1/lib/google/cloud/bigquery/service.rb:525:in `rescue in execute'
    /usr/local/bundle/gems/google-cloud-bigquery-1.21.1/lib/google/cloud/bigquery/service.rb:518:in `execute'
    /usr/local/bundle/gems/google-cloud-bigquery-1.21.1/lib/google/cloud/bigquery/service.rb:100:in `insert_dataset'
    /usr/local/bundle/gems/google-cloud-bigquery-1.21.1/lib/google/cloud/bigquery/project.rb:1007:in `create_dataset'
    /tmpfs/src/github/ruby-docs-samples/bigquery/snippets/acceptance/helper.rb:26:in `create_temp_dataset'
    /tmpfs/src/github/ruby-docs-samples/bigquery/snippets/acceptance/load_table_gcs_csv_test.rb:21:in `block (2 levels) in <top (required)>'

I tried to rerun the test with adding a label kokoro:force-run, but it doesn't work.

@jiren Can I merge this in?

@jiren Can I merge this in?

Yes,  Please.

authors: hengfengli jiren

repo: GoogleCloudPlatform/ruby-docs-samples


text:
fix: Update README.md

Added some blanket text to build on for this repo.
fixes #17

/gcbrun


text:
OpenTF support

https://opentf.org/announcement
We want to understand how Terraformer is affected by the OpenTF announcement, and what the plans for support are moving forward.
From a technical perspective, I expect that it will be a no-op for the time being as OpenTF anticipates maintaining compat/interop for the foreseeable future. (However, OpenTF has stated that some functionality may be added to OpenTF that is a superset of what's in base Terraform. [source])
From a process perspective:

If there is a bug in OpenTF that does not exist in Terraform (or vice-versa), will bug reports (as they pertain to this project) be treated equally on both platforms?

Will testing be performed with code targeting both platforms?

For OpenTF's potential future "superset" functionality, will that be treated as a first-class citizen?

For those who were looking for "OpenTofu", they are the same thing:
https://opentofu.org/blog/opentofu-announces-fork-of-terraform/

opentofu just GA today, https://github.com/opentofu/opentofu/releases/tag/v1.6.0

This project should move towards OpenTofu

@sergeylanzman @meshuga @skarimo

It would seem that this project leverages the Terraform Protocol, rather than relying directly on Terraform or OpenTofu directly. As long as the protocol continues to be supported, then the providers which implement the protocol will be supported.

authors: chazzmoney chenrui333 lucasfcnunes skyzyx

repo: GoogleCloudPlatform/terraformer


text:
Add scope+include/exclude support in mojo

Description of the issue:
all dependencies of the mojo are added to the images
Expected behavior:
dependencies should be filterable, common way for maven is

scope
explicit includes/excludes (g:a:v[:t:c])

คำอธิบายของปัญหา :
การพึ่งพาทั้งหมดของโมโจจะถูกเพิ่มเข้าไปในภาพ
พฤติกรรมที่คาดหวัง :
ควรขึ้นอยู่กับการกรองกรองวิธีทั่วไปสำหรับ maven คือ

ขอบเขต
ชัดเจนรวม / ไม่รวม ( g:a:v[:t:c])

At the end it can be solved by duplicating poms but since it is 1. common in mbn plugin and 2. as trivial as a filter in a stream, i really think it is worth adding it built in. Workatound is indeed to use jib core in a main and run it with mvn exec plugin.

You don't need to duplicate poms. The usual way is to create multiple Maven profiles (dev, qa, staging, prod, etc.).
Otherwise, when it comes to the built-in filtering support, I think we can close this as a dup in favor of others for supporting fine-grained control over dependencies and layers. See
https://github.com/GoogleContainerTools/jib/issues/1962#issuecomment-529161358
https://github.com/GoogleContainerTools/jib/issues/403#issuecomment-404461239
https://github.com/GoogleContainerTools/jib/issues/403#issuecomment-405028122
https://github.com/GoogleContainerTools/jib/issues/403#issuecomment-405051107
https://github.com/GoogleContainerTools/jib/issues/403#issuecomment-405099150
https://github.com/GoogleContainerTools/jib/issues/1436#issuecomment-486772623
https://github.com/GoogleContainerTools/jib/issues/1852
https://github.com/GoogleContainerTools/jib/issues/2450#issue-612721518
We expect to support the generalized fined-grained dependency control through the Jib extension framework: https://github.com/GoogleContainerTools/jib/issues/1962#issuecomment-599235543

For reference profiles is not an option cause it requires rebuilds and is not release compatible whereas multiplying artificially modules work but slows down the build with no gain (why bypassing jib plugin to use core programmatically is saner).

@rmannibucau @natlovewy2535
As promised, the Jib Extension Framework is now available with the latest Jib versions. You can easily extend and tailor the Jib plugins behavior to your liking.
We've written a general-purpose layer-filter extension that enables fine-grained layer control, including deleting files and moving files into new layers. The extension will cover your use-case to filter unwanted dependencies (or any files).

Maven Layer-Filter Extension
Gradle Layer-Filter Extension

For general information about using and writing extensions, take a look at the Jib Extensions repo.

authors: chanseokoh natlovewy2535 rmannibucau

repo: GoogleContainerTools/jib


text:
unable to add specific image tag using jib instead of latest

Description of the issue:
when i am using jib maven plugin to push my app to ECR repo , its adding the image in ECR as below.
<awsid>.dkr.ecr.us-east-2.amazonaws.com/test/testrepo:latest
But  i want to attach my application name and specific version to the image. is there any way i can achieve this. is it a bug in jib?
i am getting error while attaching the app name and version to image :
The repository with name 'test/testrepo/myapp' does not exist in the registry
i am using below configurations in pom.xml

com.google.cloud.tools
jib-maven-plugin
1.3.0
			<configuration>

				<from>
					<image>openjdk:8u212-b04-jre-slim</image>
				</from>
				<container>
					<mainclass>test</mainclass>

				</container>

				<to>
					<image><awsid>.dkr.ecr.us-east-2.amazonaws.com/test/testrepo</image>

				</to>

			</configuration>

			<executions>
				<execution>
					<phase>package</phase>
					<goals>
						<goal>build</goal>
					</goals>
				</execution>
			</executions>
		</plugin>

Expected behavior:
i want my application image should be pushed as <awsid>.dkr.ecr.us-east-2.amazonaws.com/test/testrepo/myapp:<version>
Steps to reproduce:

Environment:
jib-maven-plugin Configuration:
PASTE YOUR pom.xml CONFIGURATION HERE

jib-gradle-plugin Configuration:
PASTE YOUR build.gradle CONFIGURATION HERE

Log output:
Additional Information:

Hi @murali474,
This should work.
The repository with name 'test/testrepo/myapp' does not exist in the registry

The error message above might be true. Since you said you can push test/testrepo:latest, why don't you try test/testrepo:sometag instead of test/testrepo/myapp:sometag?

test/testrepo:sometag this way its working

Yeah, then I think the error message is right, and I think test/testrepo/myapp:latest will fail too. I don't think the tag string is the issue here. Maybe you should create the new repository test/testrepo/myapp somehow first? (I don't know ECR much.)

I understood that i need to create ECR repo as namespace/app then it will work

Yeah, then I think the error message is right, and I think test/testrepo/myapp:latest will fail too. I don't think the tag string is the issue here. Maybe you should create the new repository test/testrepo/myapp somehow first? (I don't know ECR much.)

Ah, I see. So, in the case of test/testrepo/myapp, looks like ECR either

doesn't support the format test/testrepo/myapp with 2+ slashes; or
test/testrepo as a whole (i.e., including the first slash) will be considered a namespace.

I will close the issue, as this is resolved. Feel free to re-open if you have more feedback on this issue.


text:
Possible 401 Unauthorized when base image is referred to by a digest and manifest is cached

Observed this in #2215.

If the Jib cache is empty, Docker Hub returns 404 (and fail) in the first run. Interestingly, subsequent runs return 401. Worth understanding why.

When a base image is referred to by a digest, we first look into the manifest cache. If the manifest is in the cache, we return it along with no (i.e., null) base image credentials.
    if (buildContext.isOffline() || imageReference.isTagDigest()) {
      Optional<Image> image = getCachedBaseImage();
      if (image.isPresent()) {
        return new ImageAndAuthorization(image.get(), null);
      }

However, if the base image cache is missing some layers (e.g., pulling base image layers failed for some reason after caching the manifest), Jib will go out and try to download the missing layers. But it may fail because we didn't return base image credentials above.

Closing as fixed by layers check added in #3767.

authors: chanseokoh emmileaf


text:
AWS ECR - Registry may not support pushing OCI Manifest or Docker Image Manifest Version 2, Schema 2

Environment:

Jib version: 3.0.0
Build tool: Gradle 6.7.1
OS: Arch Linux (kernel: 5.11.15-arch1-2, 64-bit)

Description of the issue:
Docker image push to AWS ECR fails with Registry may not support pushing OCI Manifest or Docker Image Manifest Version 2, Schema 2 error. I'm using AWS ECR Credentials Helper as an authorization mechanism. adoptopenjdk:11-jdk Docker image used in environment where this plugin is used.
Expected behavior:
Plugin successfully pushes Docker image to AWS ECR.
Steps to reproduce:

Use 3.0.0 version of com.google.cloud.tools.jib
Export env vars: AWS_DEFAULT_REGION, AWS_ECR_REGISTRY_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
Install https://github.com/awslabs/amazon-ecr-credential-helper version 0.5.0-2
Build with: ./gradlew jib -Djib.to.image=${AWS_ECR_REGISTRY_URL}:latest -Djib.to.credHelper=ecr-login

jib-gradle-plugin Configuration:
jib {

}

Log output:
./gradlew jib -Djib.to.image=${AWS_ECR_REGISTRY_URL}:latest -Djib.to.credHelper=ecr-login

> Task :jib

Containerizing application to XXX.dkr.ecr.eu-north-1.amazonaws.com/YYY...
Base image 'adoptopenjdk:11-jre' does not use a specific image digest - build may not be reproducible
Using credential helper docker-credential-ecr-login for XXX.dkr.ecr.eu-north-1.amazonaws.com/YYY
The base image requires auth. Trying again for adoptopenjdk:11-jre...
Using base image with digest: sha256:944587c7efcc21dc419cb838422c23bc9c1748f75b8cc8e8cafdf1d5a555d16d

Container entrypoint set to [java, -cp, /app/resources:/app/classes:/app/libs/*, YYY]
Executing tasks:
[=========================     ] 83.3% complete
> launching layer pushers

> Task :jib FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push image manifest for XXX.dkr.ecr.eu-north-1.amazonaws.com/YYY:latest but failed because: Registry may not support pushing OCI Manifest or Docker Image Manifest Version 2, Schema 2 | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 5s
3 actionable tasks: 1 executed, 2 up-to-date

Hey @mantoshelis, apologies for the delay. This error is only thrown in one place in the Jib code and is due to the server returning MANIFEST_INVALID or TAG_INVALID. We've seen this occur in the past for some registries like Quay.io and Artifactory.  I would suggest taking a look at #3201 (comment) and the related issues linked there. I'm tempted to say that it's very similar to what you're experiencing.

The difference from #3201 is that, the build here doesn't seem to be pushing multiple manifests. Otherwise, you must have seen all the tags in the log, but it says only one image (manifest):
Containerizing application to XXX.dkr.ecr.eu-north-1.amazonaws.com/YYY...

However, it's still true that the server is returning MANIFEST_INVALID or TAG_INVALID, and I think it's an issue in AWS ECR, maybe specific to your repository for some reason. (We have not heard any single case of Jib creating and pushing an invalid manifest for many years. And people are regularly pushing to ECR without an issue.)

Thank you for the responses. Unfortunately, the problem persisted in AWS ECR registry. Tag mutability was toggled on.

Tag mutability was toggled on.

Aha, thanks for the update. That explains it.

The error is because Jib is pushing the exact same image again (i.e., ECR already has the image). With the tag immutability turned on, ECR won't allow re-uploading existing images.
Obviously, if you hit this issue, there's nothing you need to do; the image is already there. But for some reason you don't want to see Jib fail, one option could be -DskipExistingImages (e.g., ./gradlew jib -DskipExistingImages). Jib will do nothing for the images already existing in a target registry.

The skipExistingImages option does not work for me :/
This is the profile I'm using
        <profile>
            <id>jib-dev</id>
            <activation>
                <property>
                    <name>jib-dev</name>
                    <value>true</value>
                </property>
            </activation>
            <dependencies>
                <!-- https://mvnrepository.com/artifact/com.google.cloud.tools/jib-maven-plugin -->
                <dependency>
                    <groupId>com.google.cloud.tools</groupId>
                    <artifactId>jib-maven-plugin</artifactId>
                    <version>3.0.0</version>
                </dependency>
            </dependencies>
            <build>
                <defaultGoal>compile test jib:build</defaultGoal>
                <plugins>
                    <plugin>
                        <groupId>com.google.cloud.tools</groupId>
                        <artifactId>jib-maven-plugin</artifactId>
                        <version>3.0.0</version>
                        <configuration>
                            <container>
                                <entrypoint>
                                    <arg>/bin/sh</arg>
                                    <arg>-c</arg>
                                    <arg>echo "PROFILE: $PROFILE"; exec java \
                                        -Dspring.profiles.active=$PROFILE \
                                        $JAVA_OPTS -cp /app/resources:/app/classes:/app/libs/* \
                                        com.company.MyApplication
                                    </arg>
                                </entrypoint>
                            </container>
                            <from>
                                <image>
                                    <ACCOUNTID>.dkr.ecr.eu-central-1.amazonaws.com/baseimages:openjdk-8
                                </image>
                            </from>
                            <to>
                                <image>
                                    <ACCOUNTID>.dkr.ecr.eu-central-1.amazonaws.com/myapp-unstable:dev
                                </image>
                            </to>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>

I use this command to run the build mvn -B -Djib-dev=true -Djib.skipExistingImages

@chtpl for documentation purposes: -Djib.skipExistingImages skips pushing images that already exist in a registry (i.e., when trying to push the same image). The "same image" here means the image with the exact same contents, which has the same image digest (SHA). -Djib.skipExistingImages doesn't skip pushing a different image only because a registry already has the same image tag.

authors: chanseokoh chtpl mantoshelis mpeddada1


text:
Main class was not found for Spring Boot 3.1 and 3.2

Environment:

Jib version: 3.4.0
Build tool: maven 3.9.5
OS: OL 8

Description of the issue:
When building my application the build fails with
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (default-cli) on project helloworld-31: Main class was not found, perhaps you should add a mainClass configuration to jib-maven-plugi
Expected behavior:
JIB should be able to introspect and find the right MainClass for Spring Boot applications.
If I force set the main class using
props.setProperty("jib.container.mainClass", "org.springframework.boot.loader.JarLauncher");
props.setProperty("jib.container.mainClass", "org.springframework.boot.loader.launch.JarLauncher");

I can get it to work however the MainClass for Spring Boot depends on the version. But that is not a very elegant way :-)
jib-maven-plugin Configuration:
            <build>
                <plugins>
                    <plugin>
                        <groupId>com.google.cloud.tools</groupId>
                        <artifactId>jib-maven-plugin</artifactId>
                        <version>3.4.0</version>
                        <configuration>
                            <containerizingMode>packaged</containerizingMode>
                            <allowInsecureRegistries>true</allowInsecureRegistries>
                        </configuration>
                    </plugin>
                </plugins>
            </build>

and I add this in Java as I build the image in an application and not from the command line.
        props.setProperty("interactiveMode", "false");
        props.setProperty("jib.from.image", "ghcr.io/graalvm/jdk:ol7-java17-22.2.0");
        props.setProperty("jib.to.image", registryUrl + "/" + imageName + ":" + imageVersion);
        props.setProperty("jib.to.auth.username", registryUsername);
        props.setProperty("jib.to.auth.password", registryPassword);
        props.setProperty("jib.container.format", "OCI");

Thanks
Andy

Some more information, the MANIFEST.MF file contains the MainClass value (different for different version of SB).
Spring Boot 3.2
Manifest-Version: 1.0
Created-By: Maven JAR Plugin 3.3.0
Build-Jdk-Spec: 17
Implementation-Title: helloworld
Implementation-Version: 0.0.1-SNAPSHOT
Main-Class: org.springframework.boot.loader.launch.JarLauncher
Start-Class: com.example.helloworld.HelloworldApplication
Spring-Boot-Version: 3.2.0
Spring-Boot-Classes: BOOT-INF/classes/
Spring-Boot-Lib: BOOT-INF/lib/
Spring-Boot-Classpath-Index: BOOT-INF/classpath.idx
Spring-Boot-Layers-Index: BOOT-INF/layers.idx

Spring Boot 3.1
Manifest-Version: 1.0
Created-By: Maven JAR Plugin 3.3.0
Build-Jdk-Spec: 17
Implementation-Title: helloworld-31
Implementation-Version: 0.0.1-SNAPSHOT
Main-Class: org.springframework.boot.loader.JarLauncher
Start-Class: com.example.helloworld31.Helloworld31Application
Spring-Boot-Version: 3.1.6
Spring-Boot-Classes: BOOT-INF/classes/
Spring-Boot-Lib: BOOT-INF/lib/
Spring-Boot-Classpath-Index: BOOT-INF/classpath.idx
Spring-Boot-Layers-Index: BOOT-INF/layers.idx

Seems that GradleProjectProperties class has an implementation where it looks for Main-Class:
Object value = jarTask.getManifest().getAttributes().get("Main-Class");
vs the MavenProjectProperties class doesn't, well it looks for something I just don't know what?
Not sure if that is of any help?

It's been a long while, so I don't remember the specifics, but given that you are talking about JarLauncher, I think you are looking at the Spring Boot fat jar created by the Spring Boot plugin? I believe Jib doesn't use the fat jar but the thin jar created by the maven-jar-plugin. Can't you just set the main class to com.example.helloworld.HelloworldApplication (in either  <archive><manifest><mainClass> of maven-jar-plugin or jib.container.mainClass?

If I set the jib.container.mainClass to class com.example.helloworld.HelloworldApplication I get the following error when trying to run the JAR file:
Error: Could not find or load main class com.example.helloworld.HelloworldApplication
Caused by: java.lang.ClassNotFoundException: com.example.helloworld.HelloworldApplication

if I set jib.container.mainCLass to org.springframework.boot.loader.launch.JarLauncher or org.springframework.boot.loader.launch.JarLauncher (depending on the SB version) I can get it to work but setting this depending on the version of SB seems wrong?
The application is deploying a SB application and deploys it onto a k8s cluster. And the application doesn't know the SB version.

I guess it should be com.example.helloworld31.Helloworld31Application? And what if you don't set <containerizingMode>packaged</containerizingMode>?

I got two different applications, one for SB 3.1 and one for SB 3.2.
3.2 starting class is com.example.helloworld.HelloworldApplication and the one for SB 3.1 is com.example.helloworld31.Helloworld31Application
<containerizingMode>packaged</containerizingMode> is set:
            <build>
                <plugins>
                    <plugin>
                        <groupId>com.google.cloud.tools</groupId>
                        <artifactId>jib-maven-plugin</artifactId>
                        <version>${jib-maven-plugin.version}</version>
                        <configuration>
                            <containerizingMode>packaged</containerizingMode>
                            <allowInsecureRegistries>true</allowInsecureRegistries>
                        </configuration>
                    </plugin>
                </plugins>
            </build>

I'm looking at the jib-gradle-plugin and the method getMainClassFromJarPlugin() in the GradleProjectProperties class and it seems to do a very different thing, maybe it is correct, but it looks for the value "Main-Class" in the JAR file:
Object value = jarTask.getManifest().getAttributes().get("Main-Class");
vs the same mermaid in the jib-maven-plugin where the method getMainClassFromJarPlugin() is doing something very different.
Is there a DEBUG flag that I can turn on to get more data?

seems to do a very different thing

Basically it does the same thing; it retrieves the Main-Class attribute to be set inside the thin jar created by maven-jar-plugin or the Jar task. (not by the Spring Boot plugin). In Maven, what it does is
  public String getMainClassFromJarPlugin() {
    Plugin mavenJarPlugin = project.getPlugin("org.apache.maven.plugins:maven-jar-plugin");
    if (mavenJarPlugin != null) {
      return getChildValue(
              (Xpp3Dom) mavenJarPlugin.getConfiguration(), "archive", "manifest", "mainClass")
          .orElse(null);

You seem to believe that the Gradle opens up the built jar and inspect the manifest, but what it does is just to get the Main-Class information from the Jar task.
I think you should be able to see log messages by MainClassResolver where Jib got the main class info (e.g., here) when you increase the logging level to INFO or more.

I'll try to unset the <allowInsecureRegistries>true</allowInsecureRegistries>. Stay tuned

<allowInsecureRegistries>false </allowInsecureRegistries> gives the same error:
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (default-cli) on project helloworld: Main class was not found, perhaps you should add a mainClass configuration to jib-maven-plugin
Can I set the logging level using a parameter somehow?

I believe mvn -X.

Oh, and it's not allowInsecureRegistries but containerizingMode.

Ah, sorry about that!
Debugging gives me this:
[DEBUG]   (f) project = MavenProject: oracle.obaas:helloworld:0.0.1 @ /app/upload-dir/application/helloworld/containerPOM.xml
[DEBUG]   (f) session = org.apache.maven.execution.MavenSession@180b3819
[DEBUG] -- end configuration --
[DEBUG] Using JAR: /app/upload-dir/application/helloworld/target/helloworld-0.0.1.jar
[DEBUG] Searching for main class... Add a 'mainClass' configuration to 'jib-maven-plugin' to improve build speed.
[DEBUG] Could not find a valid main class from 'maven-jar-plugin'; looking into all class files to infer main class.
[DEBUG] MainClassFinder: /app/upload-dir/application/helloworld/target/classes is not a regular file; skipping

Not sure what that means?

If I don't use containerizingMode I still get this:
DEBUG]   (f) project = MavenProject: oracle.obaas:helloworld:0.0.1 @ /app/upload-dir/application/helloworld/containerPOM.xml
[DEBUG]   (f) session = org.apache.maven.execution.MavenSession@1e8ab90f
[DEBUG] -- end configuration --
[DEBUG] Searching for main class... Add a 'mainClass' configuration to 'jib-maven-plugin' to improve build speed.
[DEBUG] Could not find a valid main class from 'maven-jar-plugin'; looking into all class files to infer main class.
[DEBUG] MainClassFinder: /app/upload-dir/application/helloworld/target/classes is not a regular file; skipping
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.092 s
[INFO] Finished at: 2023-12-14T17:37:56Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (default-cli) on project helloworld: Main class was not found, perhaps you should add a `mainClass` configuration to jib-maven-plugin

Do you have a .class file compiled from public static void main(...) under /app/upload-dir/application/helloworld/target/classes? Isn't it empty? It should output logs for all the sub-directories like this:
DEBUG] MainClassFinder: /usr/local/google/home/chanseok/workspace4E/jib/examples/helloworld/target/classes is not a regular file; skipping
[DEBUG] MainClassFinder: /usr/local/google/home/chanseok/workspace4E/jib/examples/helloworld/target/classes/example is not a regular file; skipping
[DEBUG] MainClassFinder: /usr/local/google/home/chanseok/workspace4E/jib/examples/helloworld/target/classes/world is not a class file; skipping

MainClassFinder.find() is called from here and output these logs here.

I think so, I got this:
x BOOT-INF/
x BOOT-INF/classes/
x BOOT-INF/classes/com/
x BOOT-INF/classes/com/example/
x BOOT-INF/classes/com/example/helloworld/
x META-INF/maven/
x META-INF/maven/com.example/
x META-INF/maven/com.example/helloworld/
x BOOT-INF/classes/com/example/helloworld/HelloWorldController.class
x BOOT-INF/classes/com/example/helloworld/HelloworldApplication.class
x BOOT-INF/classes/application.properties
x META-INF/maven/com.example/helloworld/pom.xml
x META-INF/maven/com.example/helloworld/pom.properties

Attached is the JAR file that is used [DEBUG] Using JAR: /app/upload-dir/application/helloworld/target/helloworld-0.0.1.jar
helloworld-0.0.1-SNAPSHOT.jar.zip

I am talking about /app/upload-dir/application/helloworld/target/classes (your log may be when not setting containerizingMode), not the contents inside the jar.

when not setting containerizingMode I get this error:
[DEBUG]   (f) project = MavenProject: oracle.obaas:helloworld:0.0.1 @ /app/upload-dir/application/helloworld/containerPOM.xml
[DEBUG]   (f) session = org.apache.maven.execution.MavenSession@1e8ab90f
[DEBUG] -- end configuration --
[DEBUG] Searching for main class... Add a 'mainClass' configuration to 'jib-maven-plugin' to improve build speed.
[DEBUG] Could not find a valid main class from 'maven-jar-plugin'; looking into all class files to infer main class.
[DEBUG] MainClassFinder: /app/upload-dir/application/helloworld/target/classes is not a regular file; skipping

And the /app/upload-dir/application/helloworld/target/classes is empty or at least I can't see them, maybe they get deleted before I can see them.
bash-4.4# ls -l
total 21628
drwxr-xr-x. 2 root root        6 Dec 14 19:02 classes
-rw-r--r--. 1 root root 22143943 Dec 14 19:02 helloworld-0.0.1.jar
bash-4.4# ls -l classes
total 0

Maybe a clarification on what we're trying to helps (or maybe not):

We ask our customers to build a Spring Application and build a fat Spring JAR file of the application (spring-boot-maven-plugin)
We then user JIB to package and containerize the JAR file. We are trying to avoid the (well known) security challenges of using a docker daemon during the build process.
We then build the container and deploy it to an already installed k8s cluster (same for both build and deploy)

Thanks. As you said, I believe /app/upload-dir/application/helloworld/target/classes is really empty, because otherwise, you should have seen some logs about its sub-directories as in my test project. And that's why Jib can't automatically find the main class. I think I see the picture now.
First of all, Jib doesn't use the fat jar produced by the spring-boot-maven-plugin. Whether they use the Spring Boot plugin, it doesn't matter. Jib is designed to pick up the thin jar by the maven-jar-plugin.
Then, normally, you should see two jars under target/:

fat jar by spring-boot-maven-plugin: helloworld-0.0.1.jar
thin jar by maven-jar-plugin: helloworld-0.0.1.jar.original

This no longer may be true if things might have changed. But this is what I remember from the past. And given the circumstances, this is my guess:
You just copied or left only the fat jar and are tricking Jib into believing that helloworld-0.0.1.jar is the only jar to be able to containerize the fat jar. That may be why you set <containerizingMode>packaged, even though unpacked containerizing works just fine. And hopefully you are also pretending that your Maven project has no dependencies. (If you are not, you'll be duplicating dependency jars (those put by Jib and those inside the far jar), which is super bad.
That's why manually setting the main class to com.example.helloworld31.Helloworld31Application. The class is not accessible as a main class in the fat jar. It must be JarLauncher. And since JarLauncher is not a project class but some random class from one of the project dependencies, so Jib can never infer it. And your Maven project tells Jib that there exist no classes.
If all these are true, I'd say it's kind of a weird hack. If you have to use a Spring Boot fat jar (which I think you managed to achieve it with some tricks), then it's a fair game. You should set a correct main class yourself.

But I discourage using a fat jar, because it completely destroys a lot of goodness about layering and reproducibility.

If I use helloworld-0.0.1-SNAPSHOT.jar.original I still get the same error:
[DEBUG]   (f) project = MavenProject: oracle.obaas:helloworld:0.0.1 @ /app/upload-dir/application/helloworld/containerPOM.xml
[DEBUG]   (f) session = org.apache.maven.execution.MavenSession@1e8ab90f
[DEBUG] -- end configuration --
[DEBUG] Searching for main class... Add a 'mainClass' configuration to 'jib-maven-plugin' to improve build speed.
[DEBUG] Could not find a valid main class from 'maven-jar-plugin'; looking into all class files to infer main class.
[DEBUG] MainClassFinder: /app/upload-dir/application/helloworld/target/classes is not a regular file; skipping
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.829 s
[INFO] Finished at: 2023-12-14T19:46:24Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (default-cli) on project helloworld: Main class was not found, perhaps you should add a `mainClass` configuration to jib-maven-plugin

I'll find some way to make it work

Yeah, that's because /app/upload-dir/application/helloworld/target/classes. However, when using the thin jar, I think manually setting the main class (either in maven-jar-plugin or in Jib) to HelloWorld31Application will probably work.

It doesn't work

authors: andytael chanseokoh


text:
ci: create junit report for unit tests

This generalizes the junit wrapper so that it can be ran for both the
e2e tests and the unit tests in a similar way. This is intended to make
it easier to digest test results in the prow UI.
Based on https://github.com/GoogleContainerTools/kpt-config-sync/pull/997 for the common installation path of the local junit-report binary

Doesn't look like the report is getting rendered quite right, need to do a bit more digging
/hold

Looks like the issue was that you need to pass the -v flag to go test to get the right output format.
/retest
/unhold

repo: GoogleContainerTools/kpt-config-sync


text:
make retagging functionality generic #99

This change updates the make targets associated with retagging images to make them more generic/reusable.

/hold
Rebase and unhold until https://github.com/GoogleContainerTools/kpt-config-sync/pull/163 is merged.

/unhold

/lgtm
/approve

/retest

authors: nan-yu sdowell


text:
Stream e2e log output

Streaming requires test verbosity and one package at a time, but still allows any number of parallel tests.

/lgtm
/approve

authors: karlkfi sdowell


text:
Add copy button to catalog.

Fixes https://github.com/GoogleContainerTools/kpt/issues/2128

@mengqiy Could you audit the catalog docs for rule 2: https://github.com/GoogleContainerTools/kpt/blob/main/docs/style-guides/docs.md
Otherwise copy functionality doesn't work.

Could you audit the catalog docs for rule 2

Sure

Fixed markdown code blocks in https://github.com/GoogleContainerTools/kpt-functions-catalog/pull/383

authors: etefera frankfarzan mengqiy


text:
Document symlinking and underscoring

We should explain:

the new bin path introduced  (in architecture guide?)
we do symlinking (in architecture guide?, possibly dev guide?)
we automatically convert -s in plugin names to _ (dev guide?)

Done in #100.

repo: GoogleContainerTools/krew


text:
Close reason for PaginatorInterface

The Problem
I'd like to delete the invoking command when a user uses the close button on a PaginatorInterface.
As I do not use the delete_message option I don't want to unconditionally do this when the interface is done.
The Ideal Solution
Have some way of knowing whether the interface was closed by the user, or was closed via the timeout.
interface = PaginatorInterface(...)
await interface.send_to(channel)

await interface.task

if interface.closed_by_user:  # How exactly this looks doesn't really matter to me
    await ctx.message.delete()

The Current Solution
One could wait for the message to be deleted, however this seems rather impractical if used in more than one location:
interface = PaginatorInterface(...)
await interface.send_to(channel)

try:
    # This has the issue of not even working properly as the timeout counts from last button use
    await bot.wait_for('raw_message_delete', check=lambda x: x.message_id = interface.message.id, timeout=interface.timeout)
except asyncio.TimeoutError:
    return

await ctx.message.delete()

Summary
Knowing whether a PaginatorInterface was closed manually or via the timeout.

This is implemented in 3a027993ac69b271d300e6ec6663c7b870b27641 (and thus will be part of v2) as PaginatorInterface.close_exception.
It should be now possible to do the following:
if interface.closed:
    if interface.close_exception is None:
        # natural closure, e.g. by user pressing the close button
        ...
    elif isinstance(interface.close_exception, asyncio.TimeoutError):
        # closure through timeout
        ...
    elif isinstance(interface.close_exception, asyncio.CancelledError):
        # closure through cancellation, e.g. jsk cancel or the event loop closing
        ...

Thank you very much!

authors: Gorialis SnowyLuma

repo: Gorialis/jishaku

license: MIT


text:
Description property added to composer.json

./composer.json is valid for simple usage with composer but has strict errors that make it unable to be published as a package:
See https://getcomposer.org/doc/04-schema.md for details on the schema
description : The property description is required

Sure this should be a fix for #6

repo: GrKamil/laravel-telegram-logging


text:
UofT-DSI | production - Assignment_1

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)
Adding lagged features for Close and Adj_Close, calculating daily returns, creating a high-low range feature, and a 10-day rolling average.
What did you learn from the changes you have made?
I learned how to handle Dask’s metadata requirements, sort data for lag calculations, and filter valid file paths when loading .parquet files.
Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?
Yes, converting the Dask DataFrame to Pandas for simpler calculations, but it was only considered if memory was allowed due to data size.
Were there any challenges? If so, what issue(s) did you face? How did you overcome it?
Yes. It took too long to test codes. Delete the compute of the result of returns and hi_lo_range calculation (don't convert to pandas).
How were these changes tested?
Testing involved checking outputs with .head(), confirming file paths, and comparing with a small Pandas subset for accuracy.
A reference to a related issue in your repository (if applicable)
N/A
Checklist

[x] I can confirm that my changes are working as intended

a1: Complete

authors: GraceMengtingWang Melokazz

repo: GraceMengtingWang/production


text:
New feature: 'theme' dependency scope

Added 'theme' dependency scope as described in this feature request: https://github.com/GradleFx/GradleFx/issues/23
I haven't used git or GitHub all that much and I am completely new to forking and issuing pull requests, so tell me whether my approach is OK here. Perhaps I should have worked on the develop branch?

Thanks for implementing this feature! Concerning the pull request, I'm afraid you'll have to do the pull request again and point to the GradleFx develop branch and then we'll have to see whether it will merge automatically.

I guess that should work; I can think of no reason it would clash. I'll close this one then.
But this brings me to another question: today I created a localization convention (so that one can define locales=["nl_BE"] on the project and be done with it). I've done this from the master branch and I know for a fact that it will collide with some changes on the develop branch, because it treats an issue similar to https://github.com/GradleFx/GradleFx/issues/17. I guess the best approach would be to just start over on the develop branch, wouldn't it?

That would indeed be the easiest for me. All new development is done on the develop branch and is only merged to main when I'm releasing a new version.

authors: RIAstar SlevinBE

repo: GradleFx/GradleFx