text:
🛑 ojolink is down

In b24a63e, ojolink (ojolink.fr) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Ojolink is back up in c914d86.


text:
🛑 ojolink is down

In 934e551, ojolink (ojolink.fr) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Ojolink is back up in 40869aa.

source: gharchive/issue

license: MIT

license_type: permissive

license_source: github-api


text:
🛑 ojolink is down

In 629dde0, ojolink (ojolink.fr) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Ojolink is back up in e1686ba.


text:
🛑 carlinha is down

In ac9c1e1, carlinha (carlinha.org) was down:

HTTP code: 0
Response time: 0 ms

Resolved: carlinha is back up in 15ef96d after 1 hour, 48 minutes.


text:
🛑 asgreenas is down

In 4a1b821, asgreenas (asgreenas.com) was down:

HTTP code: 403
Response time: 882 ms

Resolved: asgreenas is back up in d849c1d after 14 minutes.


text:
🛑 hcr is down

In 1a21838, hcr (hcr.co.uk) was down:

HTTP code: 429
Response time: 930 ms

Resolved: hcr is back up in 7f106ee after .


text:
🛑 ifight365 is down

In c1fd636, ifight365 (ifight365.com) was down:

HTTP code: 0
Response time: 0 ms

Resolved: ifight365 is back up in 14c4863 after 20 hours, 9 minutes.


text:
🛑 ojolink is down

In 0ccc5ea, ojolink (ojolink.fr) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Ojolink is back up in 06f860b after 9 minutes.


text:
🛑 ojolink is down

In e82e47a, ojolink (ojolink.fr) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Ojolink is back up in 385521e after 12 minutes.


text:
🛑 ojolink is down

In 123ae86, ojolink (ojolink.fr) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Ojolink is back up in 699f062 after 7 minutes.


text:
🛑 ojolink is down

In b10ad48, ojolink (ojolink.fr) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Ojolink is back up in df8dac1 after 17 minutes.


text:
🛑 orkut is down

In 36a9957, orkut (orkut.co.in) was down:

HTTP code: 0
Response time: 0 ms

Resolved: orkut is back up in 3675a95 after 19 minutes.


text:
🛑 orkut is down

In 579e414, orkut (orkut.co.in) was down:

HTTP code: 0
Response time: 0 ms

Resolved: orkut is back up in de542cd after 8 minutes.


text:
🛑 orkut is down

In 252af8f, orkut (orkut.co.in) was down:

HTTP code: 0
Response time: 0 ms

Resolved: orkut is back up in 7aa4e4f after 8 minutes.


text:
🛑 torrentzap is down

In cf28d47, torrentzap (torrentzap.com) was down:

HTTP code: 0
Response time: 0 ms

Resolved: torrentzap is back up in 793eb05 after 2 hours, 1 minute.


text:
🛑 detonics is down

In 88ed62e, detonics (detonics.com) was down:

HTTP code: 403
Response time: 655 ms

Resolved: detonics is back up in 2b7ac70 after 20 minutes.


text:
🛑 orkut is down

In 9c864cf, orkut (orkut.co.in) was down:

HTTP code: 0
Response time: 0 ms

Resolved: orkut is back up in f7cd6a2 after 1 hour, 12 minutes.


text:
Getting error when return project response would be empty

When selecting multiple fields that no projects match with them we return error
curl --location --request POST 'https://serve.giveth.io/graphql' \
--header 'authority: dev.serve.giveth.io' \
--header 'accept: */*' \
--header 'accept-language: es,en-US;q=0.9,en;q=0.8,fa;q=0.7' \
--header 'authorization;' \
--header 'authversion: 2' \
--header 'content-type: application/json' \
--header 'origin: https://giveth-homepage.vercel.app' \
--header 'referer: https://giveth-homepage.vercel.app/' \
--header 'sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "macOS"' \
--header 'sec-fetch-dest: empty' \
--header 'sec-fetch-mode: cors' \
--header 'sec-fetch-site: cross-site' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36' \
--data-raw '{
    "operationName": "FetchAllProjects",
    "variables": {
        "limit": 15,
        "skip": 0,
        "connectedWalletUserId": 118,
        "filters": [
            "GivingBlock",
            "AcceptGiv",
        ]
    },
    "query": "query FetchAllProjects($limit: Int, $skip: Int, $sortingBy: SortingField, $filters: [FilterField!], $searchTerm: String, $category: String, $mainCategory: String, $campaignSlug: String, $connectedWalletUserId: Int) {\n  allProjects(\n    limit: $limit\n    skip: $skip\n    sortingBy: $sortingBy\n    filters: $filters\n    searchTerm: $searchTerm\n    category: $category\n    mainCategory: $mainCategory\n    campaignSlug: $campaignSlug\n    connectedWalletUserId: $connectedWalletUserId\n  ) {\n    projects {\n      id\n      title\n      image\n      slug\n      description\n      verified\n      totalDonations\n      traceCampaignId\n      reaction {\n        id\n        userId\n      }\n      totalReactions\n      adminUser {\n        name\n        walletAddress\n      }\n      updatedAt\n      organization {\n        name\n        label\n        supportCustomTokens\n      }\n      projectPower {\n        powerRank\n        totalPower\n        round\n      }\n    }\n    totalCount\n    categories {\n      name\n    }\n  }\n}"
}'

Response
{
    "errors": [
        {
            "message": "syntax error at or near \")\"",
            "locations": [
                {
                    "line": 2,
                    "column": 3
                }
            ],
            "path": [
                "allProjects"
            ],
            "extensions": {
                "code": "INTERNAL_SERVER_ERROR"
            }
        }
    ],
    "data": null
}

@maryjaf

It has been fixed

Thanks @mohammadranjbarz this issue has been resolved

authors: maryjaf mohammadranjbarz

repo: Giveth/impact-graph


text:
[Doubt] - How can I reset input file after submit?

Hey, how are you ?
I'm trying to reset my input file after a submit in angular component.
when I reset all inputs inside of form, the input text with filename keeps inside of it.
and when I removed the input text value with JS code (ex: obj_input.value = '';), the input don't work anymore.
do you know any way to reset the input file after a submit form ?

Hey, how are you ?
I resolved the problem.
I don't know if it is util to somebody else, but I triggered the changed event after to setup value = '';
and it works very well.
thanks.

repo: Gixx/mdl-components-ext

license: mit

license_source: bigquery


text:
Erreur apres installation

Apres installation quand je me connecte jai les erreurs suivantes :
Notice: Use of undefined constant  true - assumed ' true' in /home1/mytchopc/public_html/gladys/PHP/class/User.class.php on line 115
Notice: Use of undefined constant  true - assumed ' true' in /home1/mytchopc/public_html/gladys/PHP/class/User.class.php on line 115
Warning: Cannot modify header information - headers already sent by (output started at /home1/mytchopc/public_html/gladys/PHP/class/User.class.php:115) in /home1/mytchopc/public_html/gladys/PHP/class/User.class.php on line 118
Warning: Cannot modify header information - headers already sent by (output started at /home1/mytchopc/public_html/gladys/PHP/class/User.class.php:115) in /home1/mytchopc/public_html/gladys/PHP/connexion.php on line 8

Bizarre pour les warnings, je ne les ai pas chez moi... De toute façon une nouvelle version de Gladys est sur les rails, entièrement réécrite en Node.js ;) J'espère pouvoir la sortir rapidement !

je suspecte que c'est lie a la BD , je l'ai donc supprime et installe a nouveau , meme resultat

C'est en allant sur quelle page que tu as ce problème ?
Le soucis est juste qu'apparemment du HTML est déjà envoyé lorsqu'on essaie d'enregistrer un cookie...

jai ce message depuis la page de connexion

Lorsque tu te connecte en tapant ton mot de passe + identifiant, ou juste en allant sur la page de connexion ?
Tu as bien suivi le tutoriel d'installation ? ( http://gladysproject.com/developpeur ) Tu es sur un raspberry ?

authors: GladysProject jeanfrancis

repo: GladysProject/Gladys


text:
manual: add libusb dependency for macOS

The CLI will only run if libusb is installed.

Thanks!

source: gharchive/pull-request

authors: haroldm whitequark

repo: GlasgowEmbedded/glasgow


text:
Fix asimba.xml CE template

This asimba.xml is te part of CE setup: https://github.com/GluuFederation/community-edition-setup/blob/master/templates/asimba.xml
We need to fix is to provides default minimal configuration. If it possible we need to configure it to use local LDAP for user authentication.

I would say by default it should point at the Gluu Server Shibboleth IDP.

Should we introduce components dependency in setup for this case? If user select asimba should we select IDP?

We uses gluu-Asimba now

authors: nynymike yurem

repo: GluuFederation/oxAsimba


text:
Client listening mode support

Hey, thanks for the useful library so far!
Are there any plans for supporting listening mode（reverse）？
like this
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/installation_guide/sn-remoteaccess-installation-vnclistener
28.2.2. Connecting the Installation System to a VNC Listener
To have the installation system automatically connect to a VNC client, first start the client in listening mode. On Red Hat Enterprise Linux systems, use the option to run vncviewer as a listener. In a terminal window, enter the command: -listen
vncviewer -listen
Note
By default, vncviewer uses TCP port 5500 when in listening mode. The firewall must be configured to permit connections to this port from other systems. Choose System → Administration → Firewall. Select Other ports, and Add. Enter 5500 in the Port(s) field, and specify tcp as the Protocol.
Once the listening client is active, start the installation system and set the VNC options at the prompt. In addition to and options, use the option to specify the name or IP address of the system that has the listening client. To specify the TCP port for the listener, add a colon and the port number to the name of the system. boot:vncvncpasswordvncconnect
For example, to connect to a VNC client on the system on the port 5500, enter the following at the prompt: desktop.mydomain.comboot:
linux vnc vncpassword=qwerty vncconnect:5500=desktop.mydomain.com

Haven't checked it out yet (didn't even know this existed 😉 ). Currently there is not plan to implement it. I'm not even sure if this is the correct package to implement it or flutter_rfb would be better. Or the part of the protocol here and the client/listen logic part in flutter_rfb. I'd have to dive deeper into the reference docs to first understand how this whole thing works.
But if you really need it, feel free to invest the time and create a PR. I'll happily review it.

authors: Goddchen jaychouwii

repo: Goddchen/dart-rfb


text:
Simple support for using large or unbounded sets of textures over time

This PR fixes two issues when loading a large number of textures over time. For example, our use case is browsing massive image stacks of microscopy data, so while the size of textures in use at any given time is reasonable, a browsing session over several hours may cover dozens of GB of image data.

The ImageLoader for each texture is retained, which can accumulate memory and crash the browser.
The default texture caching strategy is cumulative, which while appropriate for games with a known, in-memory limit on the total number of textures required, does not fit our use case. Again this results in an OOM browser crash.

The current implementation solving 2, creating a LRU cache manager wrapping BaseTextureCache, is a bit rough, but I wanted to gauge interest in merging before cleaning it up since it meets our needs as is.

Interesting changes, sounds like you are not destroying the base textures when you are done with them though (which removes them from the cache).
https://github.com/GoodBoyDigital/pixi.js/blob/v2.x/src/pixi/textures/BaseTexture.js#L166
You will want to do myTexture.destroy(true); so the base texture is destroyed as well and both are cleaned out of the cache.
I like the .once() update, but I'm not conviced the LRU is the way to do this. Or more importantly that it seems out of Pixi's scope.
(P.S. you should upgrade to v3, it is easy and many app even "just work" and warn you on what to change).

Right, the textures are not destroyed because they may be quickly needed again if the user browses back over that part of the image stack (i.e., which textures will be reused has some locality but is not predictable), hence the need for a caching manager. But I agree it's probably outside of Pixi's scope, and it's easy to hack this into Pixi for those who do need it.
Should I close this PR and open one with just the ImageLoader fix?
We've been very happy with Pixi and will definitely upgrade to v3 in a future relase. Thanks for the great software.

Hey guys - this all looks quite interesting. My two pence.. this could make a nice module / plugin for pixi?

authors: GoodBoyDigital aschampion englercj


text:
User name is missed on the Dashboard

Pre-condition: User is logged out from wallet
Steps to reproduce:

Open the https://goodqa.netlify.com/
Sign in using mnemonic
Pay attention on the user's name

Actual result:  User's name isn't displayed
Expected result: User's name is display correctly
Environment: QA
User's mnemonic: before guitar melt always glimpse erupt sugar repair replace style chimney caution
Video: https://www.dropbox.com/s/mmf60l0g8urgpi6/2019-12-04_13-03-58.mp4?dl=0

@sirpy @AnastasiiaOdnoshevna
I just take a look at the user records in the databases for provided mnemonic and found that the gun record is almost empty: only loginToken and marketToken fields exist for that user record (which likely fetched from server on dashboard did mount - prepareLoginToken), but the mongo record is complete/fulfilled (no field is missed, everything fine).
After discussing this issue with the team we didn't find the way of how it could happen.
In order to prevent such a situation from occurring in the future, we propose to add additional restore account flow (in the scope of a new task) which will check if there is something wrong with the user record in the gun and if yes then try to restore an account from the server/mongo record.

@sirpy I think this issue could be related to the gun-indexedDB setup issue too.

@AnastasiiaOdnoshevna  Please check again. It should be fixed with "gun.get returns soul on first query"

repo: GoodDollar/GoodDAPP


text:
[RN] (BUG) #2315 claim success popups are in different sizes

Description
Its a copy of #2366 PR for react-native branch.

@yaroslav-fedyshyn-nordwhale I think that this PR is subscribing all changes

ok. I will create a new PR with related fixes only. I think something went wrong when merging react-native into this PR.

Closing this PR as a new one created with related changes only - https://github.com/GoodDollar/GoodDAPP/pull/2401


text:
When user enters text into "other" text field, check the accompanying button/box

For #86.

@lgarron another trivial change

#86 is about images... the wrong reference, perhaps?

Whoops. I meant #68 !

LGTM

authors: adrifelt lgarron moduloprime

repo: GoogleChrome/experience-sampling


text:
A way to shutdown ServiceWorker

It would be good if there was a way to shutdown/stop ServiceWorker. This is needed for tests.

Since we are waiting for #31 you can use at this moment one of Chrome DevTools Protocol - ServiceWorker Domain.
For example:
await page._client.send('ServiceWorker.enable');
await page._client.send('ServiceWorker.stopAllWorkers');
// or
await page._client.send('ServiceWorker.unregister', { scopeURL: "http://your-url" });

@Everettss thanks for a suggestion, sounds good!

Closing as a dupe of #2634

authors: Everettss NekR aslushnikov

repo: GoogleChrome/puppeteer


text:
Generate the pdf with empty content occasionally

Steps to reproduce
Tell us about your environment:

Puppeteer version: v1.0.0
Platform / OS version: Darwin64
URLs (if applicable): https://dev.capitalpreferences.com/dashboard/setup/signin
Node.js version:
v7
What steps will reproduce the problem?
While I tried to load the url and generate the pdf, it only has PDF with content generated first time. Always blank pdf for consecutively call.
Please include code that reproduces the issue.

What is the expected result?
What happens instead?

Any update on this?

@PCS-Ismail im in the same boat where the pdf will return empty occasionally. I believe it only happens on heroku for me

Happens on Kubernetes for me, either empty or the pdf is incomplete. I'll follow through on this

+1

Thanks, @vc1 waiting for domcontentloaded worked for me!

I had similair problem:
My button text and the text itself disappeared sometimes from pdf,
adding:
await page.waitFor('*') ,
solved my problem.
this waits every element in the html of the page.

same empty bugs

empty content occasionally

I'm using Angular Server Side Rendering also got the same problem. The await page.waitFor('*')  is not work at my side. I must set a static delay value before generating PDF to fix this problem. Look likes the issue come from the Angular SSR, it's re-render at Client Side after Server Side have responsed
 await page.waitFor(5000);

 await page.pdf({
      format: 'A4',
      printBackground: true,
      path: filePath1
 });

authors: Benew Faisalnwz01 PCS-Ismail TonyWuLihu maxmax1992 phamcharles phattranky xyzdata


text:
mouse down,move,up can't be trigger twice

Steps to reproduce
**Tell us about your environment:

Puppeteer version: v1.6.0
Platform / OS version: Windows 10
URLs (if applicable):
Node.js version: v8.11.1

What steps will reproduce the problem?
Please include code that reproduces the issue.
const puppeteer = require('puppeteer');
const devices = require('puppeteer/DeviceDescriptors');
const iPhone = devices['iPhone 7'];

function resolveAfter6Seconds (params) {
    return new Promise(resolve => {
        setTimeout(() => {
            console.log(params)
            resolve('resolved');
        }, 6000);
    });
}

(async () => {
    const browser = await puppeteer.launch({ headless: false })
    page = await browser.newPage();
    await page.emulate(iPhone)
    await page.goto('http://u2839250.viewer.maka.im/k/1ZQ3AEQN');

    await resolveAfter6Seconds('fitst 6s')
    await page.mouse.down()
    await page.mouse.move(100, -200)
    await page.mouse.up()
    console.log('first drag end')

    await resolveAfter6Seconds('second 6s')
    await page.mouse.down()
    await page.mouse.move(100, -200)
    await resolveAfter6Seconds('89798')
    await page.mouse.up()
    console.log('second drag end')

    await resolveAfter6Seconds('third 6s')
    await page.mouse.down()
    await page.mouse.move(100, -200)
    await page.mouse.up()
    console.log('third drag end')

    await browser.close();
})();

What is the expected result?
the second 'mouse.down-mouse.move-mouse.up' should work
the second page should be hidden, the third page show be shown
What happens instead?
the first 'mouse.down-mouse.move-mouse.up' works,
but the second doesn't

@JoelEinbinder can you please take a look?

Unfortunately this is a drag-n-drop bug. Duplicate of #1376.

authors: JoelEinbinder StoneCode5 aslushnikov


text:
setting {dumpio:true} breaks Puppeteer

Steps to reproduce
Tell us about your environment:

Puppeteer version: 1.9.0
Platform / OS version: Mac OS (High Sierra)
URLs (if applicable): N/A
Node.js version: v10.12.0

What steps will reproduce the problem?
Running the code below with the dumpio flag throws an error
var puppeteer = require('puppeteer');
(async () => {
    var browser = await puppeteer.launch({devtools:true,dumpio:true});
    var page = await browser.newPage();

    await page.goto('https://example.com/');
    page.on('console', msg => {
        console.log(msg.text());
    });
    await browser.close();
})();

Error below:
DevTools listening on ws://127.0.0.1:62864/devtools/browser/4a4737cf-5086-408a-a88f-741f31326569 [14166:775:1018/160652.576695:ERROR:delegated_frame_host.cc(174)] Not implemented reached in void content::DelegatedFrameHost::SetNeedsBeginFrames(bool) [14166:775:1018/160652.583933:ERROR:delegated_frame_host.cc(174)] Not implemented reached in void content::DelegatedFrameHost::SetNeedsBeginFrames(bool) [14166:775:1018/160653.122596:ERROR:delegated_frame_host.cc(174)] Not implemented reached in void content::DelegatedFrameHost::SetNeedsBeginFrames(bool) [14166:775:1018/160653.135915:ERROR:delegated_frame_host.cc(174)] Not implemented reached in void content::DelegatedFrameHost::SetNeedsBeginFrames(bool) [14166:775:1018/160653.164324:ERROR:delegated_frame_host.cc(174)] Not implemented reached in void content::DelegatedFrameHost::SetNeedsBeginFrames(bool)

Sometimes Chromium throws errors. All dumpio is doing is showing them to you. You can file an upstream bug against Chromium at https://crbug.com, but I wouldn't worry about it too much.

Thanks Joel,
Appreciate your help here. I ended up using await page.evaluate instead

authors: JoelEinbinder surensilva


text:
Issue when Writing NDEF to NFC Tag

I'm using the WebNFC API. I have a problem with the write function. It works but Android immediately reads the NDEF message that I have written and displays a popup.
I would like to disable this "read" when writing an NDEF message into the tag. How can I do it? I have tried to set ignoreRead to true but it doesn't help.
writer.write("Hello World", { ignoreRead: true })
	.then(() => {
    console.log("Message written.");
  }).catch(error => {
    console.log(`Write failed :-( try again: ${error}.`);
  });

BTW, if I want to set several options, shall I separate them with a comma? Example:
writer.write(message, { ignoreRead: true, signal: abortController.signal })
For information, I have seen a small error on https://web.dev/nfc/
There is:
await writer.push("Hello world", { signal: abortController.signal });
I think that it should be writer.write()

I'm using the WebNFC API. I have a problem with the write function. It works but Android immediately reads the NDEF message that I have written and displays a popup.
I would like to disable this "read" when writing an NDEF message into the tag. How can I do it? I have tried to set ignoreRead to true but it doesn't help.

You may want to run scan() and discard onreading events if your app is in a "write UI mode".
Note that { ignoreRead: true } is already the default option.
const reader = new NDEFReader();
await reader.scan();
reader.onreading = event => {
  if (shouldProcessReadingEvents) { /* ... */ }
};

var shouldProcessReadingEvents = false;
const writer = new NDEFWriter();
await writer.write("Hello World", )
console.log("Message written.");

BTW, if I want to set several options, shall I separate them with a comma? Example:
writer.write(message, { ignoreRead: true, signal: abortController.signal })

Yep!

For information, I have seen a small error on https://web.dev/nfc/
There is:
await writer.push("Hello world", { signal: abortController.signal });
I think that it should be writer.write()

Indeed. Thanks for catching this issue: https://github.com/GoogleChrome/web.dev/pull/2381/files

Thank you François

You may want to run scan() and discard onreading events if your app is in a "write UI mode".
Isn't that a pity that it is not the default behavior? When writing to a tag, one seldom needs the processing by NFC service of the NDEF freshly written.

Do you mind sharing this feedback to https://github.com/w3c/web-nfc/issues so that other web developers see this?
This repository is for the Web NFC sample, not the spec.

Ok I will

Here it is: https://github.com/w3c/web-nfc/issues/551

Thanks!

authors: OlivierGrenoble beaufortfrancois

repo: GoogleChrome/samples


text:
Add channels-to-versions.json

Example unminified output:
{
    "timestamp": "2023-05-04T07:47:07.583Z",
    "channels": {
        "Stable": {
            "channel": "Stable",
            "version": "113.0.5672.63",
            "revision": "1121455"
        },
        "Beta": {
            "channel": "Beta",
            "version": "114.0.5735.16",
            "revision": "1135570"
        },
        "Dev": {
            "channel": "Dev",
            "version": "114.0.5735.6",
            "revision": "1135570"
        },
        "Canary": {
            "channel": "Canary",
            "version": "115.0.5749.0",
            "revision": "1138853"
        }
    }
}

For now, there’s no guarantee that the versions in this JSON file have full asset availability (ok === true). This will be addressed separately after #2.

repo: GoogleChromeLabs/chrome-for-testing


text:
fix: expand viewport validation tests

These are not intended to be exhaustive; for that, we rely on WPT.
Bug: #868

probably this should have been prefix with test:

yes, sorry :( - still learning https://www.conventionalcommits.org/

authors: OrKoN thiagowfx

repo: GoogleChromeLabs/chromium-bidi


text:
Migrate to manifest version 3

This PR aims to migrate the chrome extension to manifest version 3:

Migrate to service worker, thus don't warn user if PiP is disabled (Service worker)
Use chrome.action instead of chrome.browserAction (Action API unification)
Bundle ga.js instead of using the remote version (Remotely hosted code)
Update manifest version number (Manifest version)

This update should be transparent to users.

Looks good. I should have split my PR into two

authors: beaufortfrancois nopeless


text:
CLI: Provide feature parity between PSAT and CLI

Feature Request
Description:
Provide as much parity as possible between PSAT's extension and PSAT's CLI.
Acceptance Criteria:

[ ] Filtering capabilities

So that you know, I have planned to break this task into multiple smaller ones.

Build a new generic Sidebar for navigation (currently we are using a bit specific to the extension).
Implement Sidebar in Dashboard
Build a new generic Filter and Search mechanism and view, and make it part of the table component.
Implement filter and search to CLI Dashboard.
Provide state persistence for selected filters (needs new logic to use the generic nature of filters).
Move the generic Sidebar to the extension.
Move the filter and search for the extension.

Released as a part of v0.4.0, closing this issue.

authors: amedina maitreyie-chavan mayan-000

repo: GoogleChromeLabs/ps-analysis-tool


text:
Apache Drill initialisation action bug

It seems there is a bug in the Apache Drill initialisation actions:
Following instructions in https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/drill , I have created a cluster via
gcloud beta dataproc clusters create cluster-drill \
    --region us-central1 \
    --no-address \
    --zone us-central1-c \
    --single-node \
    --master-machine-type n1-standard-4 \
    --master-boot-disk-size 500 \
    --image-version 2.0-debian10 \
    --project myproject \
    --initialization-actions 'gs://goog-dataproc-initialization-actions-us-central1/drill/drill.sh' \
    --optional-components=zookeeper

However, once the cluster is created drill is not running properly: sudo ./drillbit.sh status returns
/usr/lib/drill/drillbit.pid file is present but drillbit is not running.

If I try starting manually with sudo ./drillbit.sh start, in the log file /usr/lib/drill/log/drillbit.out I get the error message:
ERROR o.a.c.f.imps.CuratorFrameworkImpl - Ensure path
threw exception
org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCo
de = Unimplemented for /drill

However, I am able to run a standalone version of drill with
sudo /usr/lib/drill/bin/drill-embedded

Apparently it's a incompatibility version issue with Zookeeper and with the GCS connector.
To install Drill properly the previous configuration has to modified to
gcloud beta dataproc clusters create cluster-drill \
    --region us-central1 \
    --no-address \
    --zone us-central1-c \
    --single-node \
    --master-machine-type n1-standard-4 \
    --master-boot-disk-size 500 \
    --image-version 2.0-debian10 \
    --project myproject \
    --initialization-actions 'gs://goog-dataproc-initialization-actions-us-central1/drill/drill.sh' \
    --optional-components=ZOOKEEPER
    --metadata GCS_CONNECTOR_VERSION=2.0.1


text:
When using the hbase cli quickstart, 'getting' a nonexistent key yields an NPE

$  ./hbasecli.sh get pi 3141592653
...
Exception in thread "main" java.lang.NullPointerException
at com.example.bigtable.simplecli.HBaseCLI$GetCommand.run(HBaseCLI.java:386)
at com.example.bigtable.simplecli.HBaseCLI.main(HBaseCLI.java:138)

Do you want to fix this?

Deprioritized, closing. If anyone feels strongly about this, please reply/reopen! :)

authors: ianlewis lesv meredithslota


text:
README Compatibility Lib samples are out of date.

CompatabilityChecker does not seem to have methods get_self_compatibility or get_compatibility.

Sorry that the REAME is out of date. I will look into updating them.
For now, you can query a package or pair of packages directly with http://34.68.40.69/?package=<PACKAGE1>&package=<PACKAGE2>&python-version=<VERSION>.

Thank you for the quick response @ylil93!


text:
Workaround for Stackdriver dropping newlines.

This is workaround for issue #1406 , notably to fix the problem where log records with newlines \n are cut-off after the first line.  For a more long-term fix, we need to discuss & decide if we will continue to use google-fluentd (which means considering the stackdriver client library), or use the more platform neutral fluentd, where configuration efforts can be better justified.
More details in #1406

Can we have a logger wrapper that replace all the new line characters with tabs when we log the message instead of updating all these individual logs?

authors: blueandgold joecheuk

repo: GoogleCloudPlatform/forseti-security


text:
Setup wizard can deploy Forseti to a shared VPC

This PR will allow users who wish to run the Forseti Compute VM in a shared VPC to do so by passing arguments to the setup wizard. The user must specify the host project id, shared vpc name, and subnetwork name as arguments.
The generated compute instance template section networkInterfaces.network will now contain a host-project and vpc-name. The host-project can either be the host project id supplied by the user or the current project id as discovered by the script. The vpc-name can either be the vpc name supplied by the user or the name string 'default'.
The generated compute instance template section networkInterfaces.subnetwork is new. It will contain a host-project defined the same as above, region defined as it was previously, and subnetwork supplied by the user or the name string 'default'.

I also would like to see a unit test around whether the VPC or non-VPC deployment is handled correctly, but the best way to do that might involve testing the generated deployment template file, which is not in place and beyond the scope of this PR.  So, I have opened #843 to address separtely.

Thanks for making the changes!  Really awesome now!!

Review status: 0 of 6 files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.

scripts/gcp_setup/environment/gcloud_env.py, line 150 at r1 (raw file):
Previously, bgleonard wrote…
I completely agree but this follows convention with the rest of this file (see line 135 for example). Happy to change it still.

Good point.  I can make a sweep later, and clean all this up.

Comments from Reviewable

authors: bgleonard blueandgold


text:
Memory leak when running test suite via command line

When running ./gradlew clean test, a memory leak is reported. Reproducible on linux, but not on mac.
The source test of this error appears to be CloudDebugGlobalPollerTest. See exception / stack trace below:
java.lang.RuntimeException: Memory leak detected: com.google.gct.idea.testing.TestUtils$PluginMockApplication@63828d7a of class class com.google.gct.idea.testing.TestUtils$PluginMockApplication
See the cause for the corresponding Disposer.register() stacktrace:
at com.intellij.openapi.util.objectTree.ObjectTree.assertIsEmpty(ObjectTree.java:209)
at com.intellij.openapi.util.Disposer.assertIsEmpty(Disposer.java:124)
at com.intellij.openapi.util.Disposer.assertIsEmpty(Disposer.java:120)
at com.intellij.openapi.application.impl.ApplicationImpl.disposeSelf(ApplicationImpl.java:308)
at com.intellij.openapi.application.impl.ApplicationImpl.access$400(ApplicationImpl.java:91)
at com.intellij.openapi.application.impl.ApplicationImpl$5$1.run(ApplicationImpl.java:268)
at com.intellij.openapi.util.ShutDownTracker$2.run(ShutDownTracker.java:172)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:726)
at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:866)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:654)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:381)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

How are you identifying the test that is causing is this?

@Ignore the IntelliJGoogleLoginMessageExtenderTest and this seems to not happen.
However running just that test without anything else also makes things work.
./gradlew clean :google-cloud-tools-plugin:test -Dtest.single=IntellijGoogleLoginMessageExtenderTest

wat?

@loosebazooka from the stack trace you can see that there is an issue with TestUtils which is used only by BasePluginTestCase. There were 2 candidate tests (at the time of writing this issue..now there are more inheritors of this class):

CloudDebugGlobalPollerTest
CloudDebugProcessHandlerTest

I could then isolate it to CloudDebugProcessHandlerTest.
You mentioned IntelliJGoogleLoginMessageExtenderTest which also extends BasePluginTestCase - this test was created after the issue was filed and could also be problematic. The issue has to do with interaction between different tests which explains why running the test alone doesn't cause it.

My last comment is a bit misleading - I think problem is with BasePluginTestCase - the tests that implement this class seem to be order-dependent causing strange non-reproducible failures.

I'm pretty sure this is caused by the change I've done in the
BasePluginTestCase to restore the IJ created application object after the
test case has run with a mocked application (to ensure that our existing
test cases will not break if they run after a test inherited from
BasePluginTestCase). However I was not able to find where is the object
created that is not disposed properly and thus causes the teardown phase to
complain about memory leak.
On Tue, Feb 16, 2016 at 1:50 PM Etan Shaul notifications@github.com wrote:

My last comment is a bit misleading - I think problem is with
BasePluginTestCase - the tests that implement this class seem to be
order-dependent causing strange non-reproducible failures.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-intellij/issues/410#issuecomment-184824809
.

@etanshaul are you working on this?  I'm thinking of giving it to Mike as a starter bug.

Sure go for it. Andras and I did a little digging earlier and attached what
we found so far to the issue.
On Tue, Mar 8, 2016 at 11:12 AM, Patrick Flynn notifications@github.com
wrote:

@etanshaul https://github.com/etanshaul are you working on this? I'm
thinking of giving it to Mike as a starter bug.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-intellij/issues/410#issuecomment-193843286
.

Hey Mike do you want to check this out?

Sure, I'll give it a shot.

It looks like our BasePluginTestCase is indeed not cleaning up something. I've been able to isolate the problem to the setup and tearDown methods. It also seems to only manifest itself when followed by a test that uses setUp and tearDown of IntelliJ's PlatformTestCase.
If you run just these two tests below, you get the same Memory leak error.
public class TestA extends BasePluginTestCase {
    @Test
    public void test() {
        System.out.println("Test A executed.");
    }
}

public class TestB extends PlatformTestCase {
    public void test() {
        System.out.println("Test B executed.");
    }
}

Continuing to investigate...

I found a way to fix it, but I'm still not completely sure why this works.
If you add com.intellij.testFramework.TestRunnerUtil.replaceIdeEventQueueSafely(); to the top of our BasePluginTestCase.setup(), the memory leak error goes away.
I found this in the internals of the IntelliJ's test framework. More specifically, this is used in the com.intellij.testFramework.UsefulTestCase, which is the base class for a collection of other utility test classes, one of which is PlatformTestCase that we use.
I also stumbled upon PlatformLiteFixture which is also a subclass of UsefulTestCase and seems to create a MockApplicationEx, similarly to our BasePluginTestCase.
@patflynn Is there any reason why we didn't use the PlatformLiteFixture for our BasePluginTestCase other than the fact that it's JUnit3 vs our BasePluginTestCase being JUnit4?

I kind of understand what's going on in BasePluginTestCase, and I like
being able to use Junit 4, and apart from this exception that shows up in
our logs, it seems to work pretty well.  If adding that call fixes the
issue, and UsefulTestCase does the same, I have no objection to inserting
it and forgetting about it.
On Tue, Mar 15, 2016 at 10:10 AM, meltsufin notifications@github.com
wrote:

I found a way to fix it, but I'm still not completely sure why this works.
If you add
com.intellij.testFramework.TestRunnerUtil.replaceIdeEventQueueSafely();
to the top of our BasePluginTestCase.setup(), the memory leak error goes
away.
I found this in the internals of the IntelliJ's test framework. More
specifically, this is used in the
com.intellij.testFramework.UsefulTestCase, which is the base class for a
collection of other utility test classes, one of which is PlatformTestCase
that we use.
I also stumbled upon PlatformLiteFixture which is also a subclass of
UsefulTestCase and seems to create a MockApplicationEx, similarly to our
BasePluginTestCase.
@patflynn https://github.com/patflynn Is there any reason why we didn't
use the PlatformLiteFixture for our BasePluginTestCase other than the
fact that it's JUnit3 vs our BasePluginTestCase being JUnit4?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/gcloud-intellij/issues/410#issuecomment-196837944

authors: akerekes etanshaul loosebazooka meltsufin patflynn

repo: GoogleCloudPlatform/gcloud-intellij


text:
Add list of methods at top of module's page

Our docs pages can get pretty long. It would be nice to offer a quick glance of all of the methods at the top of a page, that would link to their respective ?method="" pages.

:+1:

Fixed in #545

authors: ryanseys stephenplusplus

repo: GoogleCloudPlatform/gcloud-node


text:
fix: remove discoveryengine samples

Description
Fixes #3569
Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.
Per this documentation, discovery engine has been deprecated in favor of Vertex Agent Builder.
Checklist

[x] I have followed Contributing Guidelines from CONTRIBUTING.MD
[x] Tests pass:   go test -v ./.. (see Testing)
[x] Code formatted:   gofmt (see Formatting)
[x] Vetting pass:   go vet (see Formatting)
[ ] These samples need a new API enabled in testing projects to pass (let us know which ones)
[ ] These samples need a new/updated env vars in testing projects set to pass (let us know which ones)
[ ] This sample adds a new sample directory, and I updated the CODEOWNERS file with the codeowners for this sample
[ ] This sample adds a new Product API, and I updated the Blunderbuss issue/PR auto-assigner with the codeowners for this sample
[ ] Please merge this PR for me once it is approved

@telpirion Hi, these samples are current. They are in use for Vertex AI Agent Builder. Can they please be added back in?

authors: holtskinner telpirion

repo: GoogleCloudPlatform/golang-samples


text:
testing: add test for spurious region tags in go docs

Fix #438.

This is really great.
@danicat, would you like to finish up this change? Otherwise, I'll address Tyler's comments and get this merged in.

@broady I was actually doing some changes when you posted. If you can wait a few hours I'll push my updates. I'm going to make the test process a few files at once to speed it up a little bit.

@tbpg PTAL

Hmm. I liked the relative simplicity of the previous version. Was it too slow?

LGTM. Definitely makes sense to have it be in parallel. Thank you for the background info!

Test failures caused by an unrelated network error.

authors: broady danicat tbpg


text:
Document the order of cells within a row.

Such as it is (the order is not guaranteed for column families).
This fixes #82.

PTAL.

PTAL.

PTAL.
I did not skip the ci build because Doxygen, though I expect no breakage.

repo: GoogleCloudPlatform/google-cloud-cpp


text:
PubSub: Subscription.Receive returns without an error

Client
PubSub
[[projects]]
  name = "cloud.google.com/go"
  packages = [
    "compute/metadata",
    "iam",
    "internal/optional",
    "internal/version",
    "pubsub",
    "pubsub/apiv1",
    "pubsub/internal/distribution"
  ]
  revision = "777200caa7fb8936aed0f12b1fd79af64cc83ec9"
  version = "v0.24.0"

Describe Your Environment
Alpine Docker, deployed in kubernetes in GKE
Expected Behavior
According to documentation:
// Receive calls f with the outstanding messages from the subscription.
// It blocks until ctx is done, or the service returns a non-retryable error.

Actual Behavior
Receive returns without an error after about 1h.
More info
This issue was introduced somewhere between 0.22 and 0.24. In other words, when I switch to 0.22, I cannot observe this problem.
[[projects]]
  name = "cloud.google.com/go"
  packages = [
    "compute/metadata",
    "iam",
    "internal/optional",
    "internal/version",
    "pubsub",
    "pubsub/apiv1",
    "pubsub/internal/distribution"
  ]
  revision = "056a55f54a6cc77b440b31a56a5e7c3982d32811"
  version = "v0.22.0"

Just to enusre you, that this is not a mistake on our part (e.g. cancelling the context). We are using pubsub client in our wrapper, which looks like this:
func (r *receiver) Start(ctx context.Context) error {
	err := r.subscription.Receive(ctx, r.receive)

	log := r.log.WithFields(logrus.Fields{
		"context_error":   ctx.Err(),
		"error":            err,
	})

	if err != nil {
		log.Infof("Receiver exited with error: '%v'", err)
	} else {
		log.Infof("Receiver exited")
	}

	return err
}

When Receive exists without an error we are observing logs like:
018-07-11 15:44:26.000 CEST Receiver exited {
 insertId:  "5ac6hwg1xi03vt"
 jsonPayload: {
  error:  null
  message:  "Receiver exited"
  context_error:  null
  timestamp:  "2018-07-11T13:44:26.694911319Z"
 }
 labels: {…}
 logName:  "projects/ingrid-head/logs/service-popular-pickuppoint"
 receiveTimestamp:  "2018-07-11T13:44:31.407873607Z"
 resource: {…}
 severity:  "INFO"
 timestamp:  "2018-07-11T13:44:26Z"
}

I can provide you with more details if you need them.

I tried to reproduce, but after several hours my Receive is still running.
Looking at the diffs between the two versions, I don't see what has changed that could cause this.
Looking at the code itself, I can't find a code path that would return in a nil without a context cancellation or timeout somewhere.
Could you add some logging to the pubsub client so we can narrow this down? Suggestions:

https://github.com/GoogleCloudPlatform/google-cloud-go/blob/v0.24.0/pubsub/subscription.go#L413
https://github.com/GoogleCloudPlatform/google-cloud-go/blob/v0.24.0/pubsub/subscription.go#L485
https://github.com/GoogleCloudPlatform/google-cloud-go/blob/v0.24.0/pubsub/subscription.go#L498

These are the only places in the code where a nil return is possible, as far as I can see. If you identify others, log those too and let me know.

TL;DR This is this line: https://github.com/GoogleCloudPlatform/google-cloud-go/blob/v0.24.0/pubsub/subscription.go#L485
Longer version:
I created a branch with logs you requested: https://github.com/krzysztofdrys/google-cloud-go/tree/add_logging_to_v0.24.0
I used this branch in my service:
[[projects]]
  branch = "add_logging_to_v0.24.0"
  name = "cloud.google.com/go"
  packages = [
    "compute/metadata",
    "iam",
    "internal/optional",
    "internal/version",
    "pubsub",
    "pubsub/apiv1",
    "pubsub/internal/distribution"
  ]
  revision = "c3f6dfbc38e85841914399de7df44423bdc58df9"
  source = "https://github.com/krzysztofdrys/google-cloud-go.git"

After 1h (service started at 2018-07-13T12:44:51.362143942Z and finished at 2018-07-13T13:45:02.085362886Z) I got this log:
subscription.go#L486

which comes from here: https://github.com/krzysztofdrys/google-cloud-go/commit/c3f6dfbc38e85841914399de7df44423bdc58df9#diff-b4556a07a2b7818c407e847791b06c3bR486

By the way: I will be away next week, but if you need additional debug info I think someone else from my team should be able to help you.

I have a theory. See https://code-review.googlesource.com/c/gocloud/+/30430.
It would be great if you could patch that in and see if it works.

Fixed by 275b871711d8709d68133bb0d128b3c1ea462bba.

authors: jba krzysztofdrys

repo: GoogleCloudPlatform/google-cloud-go


text:
Install a fake local grpc-google-iam-v1 to resolve the version conflict.

See https://readthedocs.org/projects/google-cloud-python/builds/4449660/ for validation of this fact (at 388b9bf2ad9b097ed559cb144bf03357695b5c18). This is a temporary hack to workaround the fact that the latest (0.10.1) gapic-google-pubsub-v1 relies on grpc-google-iam-v1 0.10.0, but they have contradicting bounds on oauth2client:

'oauth2client>=1.4.11, <2.0.0', in grpc-google-iam-v1
'oauth2client>=3.0.0, <4.0.0dev', in gapic-google-pubsub-v1

Fixes #2405.
WORTH NOTING: I am sending the PR from a branch I created on GoogleCloudPlatform. I created it here so I could test if the hack worked on RTD without having to keep behaving badly with premature merges.
/cc @bjwatson

ASIDE: @tseaver @jonparrott Any idea why pip is cool with the version conflict but setuptools isn't? (Evidence: python setup.py install fails at the "Processing dependencies" step with the conflict described above, but pip is happy to let it happen.)

@dhermes not sure - what does pip say is the collected and installed packages when you do it?

@bjwatson it's pretty easy to support a wide range of oauth2client versions, any reason to keep the bounds so tight?

Only failure was pylint on my fake directory: https://travis-ci.org/GoogleCloudPlatform/google-cloud-python/builds/163282822#L1000
Just added grpc-google-iam-v1-fake to IGNORED_DIRECTORIES and pushed over the original commit.
@tseaver Any issues with merging this while we wait for the version conflict to be resolved?

https://pypi.python.org/pypi/grpc-google-iam-v1/0.9.0 and https://pypi.python.org/pypi/grpc-google-iam-v1/0.10.1 are released.

@jonparrott Paranoia, per semver. I can relax it. What do you suggest? >=1.3.4, <4.0.0?

Best closed PR ever!
I just manually rebuilt latest (at 5a0e492b4098fe3c2df4d1821a432cbaee15edd6) at it succeeded: https://readthedocs.org/projects/google-cloud-python/builds/4453698/

authors: bjwatson dhermes jonparrott tseaver

repo: GoogleCloudPlatform/google-cloud-python


text:
Fix packaging of resources and tests

Don't include tests in the binary (wheel) distribution.
Include tests in the source distribution.
Include only .json and .proto package resources.
Include a LICENSE with every package (extremely important for downstream packagers).

(From #3182)

Broken into multiple commits for the sake of review, but should be squashed during merge.
All "Fix * package" commits are identical, just targeting a different package.

This is still against master, as far as I can tell.

Sigh.

authors: jonparrott lukesneeringer


text:
Error message when failed to create git repo is not useful and frustrating.

I see the error message, and I have no idea what to do.  The caller does not have permission.
Who is the caller (what is my auth identity)?  What are they trying to do?  Why don't they have permission?  How do I give them permission?  How do I change my identity?  How do I see which identities have permission?
At least some of these questions should be answered by the error message and include links or buttons to fix it.

I tried updating gcloud, and it didn't help:
PS C:\Users\Jeffrey Rennie> gcloud info
Google Cloud SDK [168.0.0]

The root cause:
PS C:\Users\Jeffrey Rennie> gcloud source repos create twelve-factor-demo
sourcerepo.googleapis.com is required for repository creation and is
not enabled.

Would you like to enable it? (Y/n)?  Y

Waiting for async operation operations/projectSettings.8b95c844-d72f-4567-a960-18b0ff8f2cc8 to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud service-management operations describe operations/projectSettings.8b95c844-d72f-4567-a960-18b0ff8f2cc8
Created [twelve-factor-demo].
You may be billed for this repository. See https://cloud.google.com/source-repositories/docs/pricing for details.
PS C:\Users\Jeffrey Rennie>

This is a general problem that we have, we need to ensure that all of the necessary APIs are always enabled, or ask the user to allow use to enable them. Just as the cloud console and gcloud do.

Hi Ivan,
What's your thought on this?   Do we collect all the APIs we need and enable them (or make sure they are enabled) in central place ?

Update. We should use ServiceManagement APIs to query if an API is enabled then call Enable API to enable it if user agrees.
The API to query the status of Enabled/Disabled is not released yet. ServiceManagement team is working to  make it public.

Fixed


text:
Easier pluggable sources & multiple sources on cmd line

Fixes #197.
This PR switches to an easier way to create pluggable sources as well as allowing for multiple sources on the command line if required. Following @vishh's refactoring to source factories, this is the next logical step IMO, allowing source factories to be registered & then created in  a consistent way. This makes it easier for sources to be contributed without the need to add command line flags for each new source.
Sources are configured on the command line via the flag:
--source=<registered_source_prefix>:<source_uri>?<source_options>

The --source flag can be specified multiple times if necessary.
As a concrete example, if you are using the kubernetes source you would specify:
--source=kubernetes:https://localhost:8443?insecure=true

Each source is identified with a prefix such as kubernetes or cadvisor which is registered in the SourceFactories registry in the extpoints package.
CoreOS & external cadvisor sources are added in the same way:
--source=cadvisor:external?standalone=true
--source=cadvisor:coreos?fleet-endpoint=http://myserver:4001

This PR covers all existing functionality for both coreos & external cadvisor sources as well as kubernetes source.
Once we work through this PR I would like to do the same for sinks.

Can one of the admins verify this patch?

ok to test
On Tue, Apr 7, 2015 at 12:41 PM, cadvisorJenkinsBot <
notifications@github.com> wrote:

Can one of the admins verify this patch?
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/heapster/pull/223#issuecomment-90709650
.

@jimmidyson: I like the simplicity of this PR! My only concern is that it is not user friendly. I remember you suggesting viper. May I know why you chose the url approach in this PR over the viper model?

I think that viper is complimentary to this - we can allow sources to be specified as flags & we can add in viper to be able to use config file too. Having both would be nice. If we decide to merge this then I can work on viper config in a subsequent PR.

I don't mind merging this as long as the existing mechanism continues to work. Once we add viper/config file support, we can get rid of the existing flags altogether.

@jimmidyson: integration/.jenkins.sh needs to be updated in addition to the travis yml file. Can you add a short doc that describes the new config scheme and may be add a link to it from the main README.md?
Overall LGTM! Thanks for working on this!

@cadvisorJenkinsBot: ok to test

@cadvisorJenkinsBot: retest

@cadvisorJenkinsBot: test this please

@jimmidyson: I figured out the reason for integration test failures. You will have to either include #226 or rebase once it gets merged. Also having a short doc that describes the new config model will help.

OK I'll rebase once merged & was going to add a doc today anyway. Thanks @vishh

Hey! Integration tests passed woohoo! Writing doc today I promise ;) Then ready for merge?

Awesome! Ping me once the docs are complete.

Docs are already updated!
On 18:36, Fri, 10 Apr 2015 Vish Kannan notifications@github.com wrote:

Awesome! Ping me once the docs are complete.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/heapster/pull/223#issuecomment-91631241
.

Good point. I'll update later tonight. Btw openshift bundles skydns as a service ootb.

Great. LGTM except for that one nit.

Updated.

Thanks. Can you squash the commits?

@vishh - squashed

Great! Will merge once the tests pass!

@vishh -have you managed to review this for merge?

Thanks for the reminder!

Awesome! Thanks
On 22:46, Mon, 13 Apr 2015 Vish Kannan notifications@github.com wrote:

Thanks for the reminder!
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/heapster/pull/223#issuecomment-92510333
.

authors: cadvisorJenkinsBot jimmidyson vishh

repo: GoogleCloudPlatform/heapster


text:
RFC: Kubectl use api servers

If we agree that we want to switch everything to use -api_servers (which might be wrong for anything other than kubelet) do we want to switch kubectl to use -api_servers, instead of its own -server=?

@eparis Are you still interested in this?

repo: GoogleCloudPlatform/kubernetes


text:
Make 'make clean' work with Docker not installed.

Fixes #1959.

LGTM

authors: jbeda smarterclayton


text:
Snowflake DECIMAL casting when running row validation with concat

When running row validation and there is DECIMAL(p,s) in Snowflake and NUMERIC(p,s) in BigQuery, the concat and hash will create different results due to how Snowflake handles DECIMAL.
Running test:
data-validation validate row \
  -sc bq_conn \
  -tc sf_conn \
  -tbls tst.TEST=tst.TEST  \
  --concat order_number,invoice_sum,status \
  --primary-keys order_number,account \
  -rr \
  -rbs 100

BigQuery DDL:
create table tst.TEST (
    order_number STRING,
    account STRING,
    invoice_sum NUMERIC(10,2),
    status STRING
);

Snowflake DDL:
create table tst.TEST (
    order_number STRING,
    account STRING,
    invoice_sum DECIMAL(10,2),
    status STRING
);

Example results:
Snowflake:
22777.10
BigQuery:
22777.1
In a nutshell:
If Snowflake has DECIMAL(10,5), the string representation is 10.00000 for 10 and 10.12000 for 10.12
If BigQuery has NUMERIC(10,5) (or DECIMAL), the string representation is 10 and 10.12 for 10.12

Hi,
Just for the reasons you point out - treating numeric as string is not a great idea. A better approach is to cast it into something that is meaningful for you from a business perspective. For example if the data is holding currency, you may only care about the first two digits of precision. If it is holding latitude or longitude, you will want more digits of precision. You could write a custom query validation where you custom query transforms the decimal into a string for comparison. It may also be possible to use Calculated Fields, though I am not very familiar with that feature.
Let me know if that resolves your issue.
Sundar Mudupalli

I'll have a look at this. We had a similar issue in the past with PostgreSQL conversions from decimal to string.

Better testing has highlighted that we have this same issue on MySQL too.

@henrihem thanks for highlighting this, it has identified a flaw in our tests which I've now also fixed. Hopefully the fix unblocks decimal validations for you.

authors: henrihem nj1973 sundar-mudupalli-work


text:
Disable (or adjust) dependabot.

The dependabot settings are inherited from upstream and this bot does not necessarily help for forked project like this.
The reason is that we get spam of updates of single package updated on EACH release branch (rate limited by dependabot). This causes issues:

It's not scalable to update single package in random time on random release branch especially manually. In fact we were not touching it so bot paused:

Every single update is essentially a "patch" that will make it much harder to sync e.g. patch version from Prometheus later on. It is also unsure if we should trigger expensive and time consuming release for each update or not?

I would suggest we disable the bot for now. If we need security updates on certain deps on certain release we might want to rather fix upstream OR manually patch version and only do that if absolutely necessary.
If we really want spam of PRs with dep updates and we want to all release branches to have all the latest deps (I don't think we want/need that), I would suggest we reconfigure dependabot to do batch upgrades, so it's managable.

Why? We've been merging Dependabot issues in the prometheus-engine repository. They're really useful, especially since at the end of this month, we'll need to start merging those to meet vulnerability requirements.

Yea, sorry for not explaining why, I should have updated the description. Edited it in (:

authors: TheSpiritXIII bwplotka

repo: GoogleCloudPlatform/prometheus


text:
Missing docker pull gcr.io/spark-operator/spark:v3.1.1 image

To move forward with a legacy application, I need to pull gcr.io/spark-operator/spark:v3.1.1 image
It seems that image does not exist anymore
❯ docker pull gcr.io/spark-operator/spark:v3.1.1

Maybe somebody removed it, because of:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/1800
what would be valid alternatives?

Or is there some way to build it myself?
The Dockerfile in root already requires that image
❯ docker build .
[+] Building 0.5s (4/4) FINISHED                                                  docker:desktop-linux
 => [internal] load build definition from Dockerfile                                              0.0s
 => => transferring dockerfile: 1.49kB                                                            0.0s
 => [internal] load .dockerignore                                                                 0.0s
 => => transferring context: 47B                                                                  0.0s
 => ERROR [internal] load metadata for gcr.io/spark-operator/spark:v3.1.1                         0.5s
 => CANCELED [internal] load metadata for docker.io/library/golang:1.19.2-alpine                  0.5s
------
 > [internal] load metadata for gcr.io/spark-operator/spark:v3.1.1:
------
Dockerfile:37
--------------------
  35 |     RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o /usr/bin/spark-operator main.go
  36 |
  37 | >>> FROM ${SPARK_IMAGE}
  38 |     USER root
  39 |     COPY --from=builder /usr/bin/spark-operator /usr/bin/
--------------------
ERROR: failed to solve: gcr.io/spark-operator/spark:v3.1.1: gcr.io/spark-operator/spark:v3.1.1: not found

Also encountering the same issue on both my k8s cluster:
Warning  Failed     22m (x4 over 23m)     kubelet            Failed to pull image "gcr.io/spark-operator/spark:v3.1.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/spar
k-operator/spark:v3.1.1 not found: manifest unknown: Failed to fetch "v3.1.1" from request "/v2/spark-operator/spark/manifests/v3.1.1".

and by using docker pull gcr.io/spark-operator/spark:v3.1.1:
Error response from daemon: manifest for gcr.io/spark-operator/spark:v3.1.1 not found: manifest unknown: Failed to fetch "v3.1.1" from request "/v2/spark-operator/spark/manifests/v3.1.1".

Seems like the image is no longer available?

Also encountering the same issue on both my k8s cluster:
Warning  Failed     22m (x4 over 23m)     kubelet            Failed to pull image "gcr.io/spark-operator/spark:v3.1.1": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/spar
k-operator/spark:v3.1.1 not found: manifest unknown: Failed to fetch "v3.1.1" from request "/v2/spark-operator/spark/manifests/v3.1.1".

and by using docker pull gcr.io/spark-operator/spark:v3.1.1:
Error response from daemon: manifest for gcr.io/spark-operator/spark:v3.1.1 not found: manifest unknown: Failed to fetch "v3.1.1" from request "/v2/spark-operator/spark/manifests/v3.1.1".

Seems like the image is no longer available?
Checking https://console.cloud.google.com/gcr/images/spark-operator/GLOBAL/spark and no images exist.

If is is possible, use docker pull apache/spark:v3.1.3 instead

Once we found out about the missing docker image, we've been using images generated from the https://github.com/apache/spark-docker repository without any major problems.

Once we found out about the missing docker image, we've been using images generated from the https://github.com/apache/spark-docker repository without any major problems.

There is no 3.1.1? Which is the version recommended

Is there any spark-py image which works with python2.7 for time being i migrate my code to python3 ?
gcr.io/spark-operator/spark:v3.0.0 was working fine for me.

Hi @jalvarez I am trying to setup the Spark on K8s using spark-on-k8s-operator, if we build the Apache spark docker image from https://github.com/apache/spark-docker repository then should we change mainApplicationFile parameter in one of the examples code?
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi.yaml#L27

Hi @jalvarez I am trying to setup the Spark on K8s using spark-on-k8s-operator, if we build the Apache spark docker image from https://github.com/apache/spark-docker repository then should we change mainApplicationFile parameter in one of the examples code?
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi.yaml#L27

If you build spark image from version 3.1.1, then you needn't change mainApplicationFile parameter.

Currently all the examples in https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/tree/master/examples appear to be referencing this non existing image. This implies the quick start guide is also broken.
Are there any properly working repository anywhere?

Hopefully someone will merge the PR or correct all examples + Dockerfile.

Probably not, given that this repo may soon be transferred to a new owner (the Kubeflow Org)

I am referencing the tutorial and can confirm that it complains about the missing image.

https://github.com/kubeflow/spark-operator/blob/master/examples/spark-pi.yaml

authors: AlejandroUPC Ayushri ChenYi015 Souheil-Yazji dbackeus jalvarez kenho811 oschrenk plfred vinayak-shanawad


text:
Spark operator generates timeout issues during pod creating after job execution on GKE

Summary
The Spark operator generates timeout issues during pod creating after job execution on a GKE cluster (1.12.7-gke.24).
Steps to reproduce
Install the operator
helm install incubator/sparkoperator --namespace spark --name spark-operator --set sparkJobNamespace=spark --set enable-metrics=true --set enableWebhook=true

Run a job containing a volume
apiVersion: "sparkoperator.k8s.io/v1beta1"
kind: SparkApplication
metadata:
  name: {{ item.name }}
  namespace: spark
spec:
  type: Scala
  mode: cluster
  image: "gcr.io/spark-operator/spark:v2.4.0-gcs-prometheus"
  imagePullPolicy: Always
  mainClass: nl.debijenkorf.core.SparkApplication
  mainApplicationFile: "gs://BUCKET/APPLICATION.jar"
  sparkVersion: "2.4.0"
  arguments:
    - {{ item.job }}
  volumes:
    - name: "test-volume"
      hostPath:
        path: "/spark/data"
        type: Directory
  driver:
    {..}
  executor:
    {..}

Actual results
I0702 08:24:10.298358      11 controller.go:522] Trying to update SparkApplication spark/test-volume-claim, from: [{  2019-07-02 07:26:25 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0    } {FAILED failed to run spark-submit for SparkApplication spark/test-volume-claim: Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create]  for kind: [Pod]  with name: [null]  in namespace: [spark]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:363)
	at org.apache.spark.deploy.k8s.submit.Client$$anonfun$run$2.apply(KubernetesClientApplication.scala:141)
	at org.apache.spark.deploy.k8s.submit.Client$$anonfun$run$2.apply(KubernetesClientApplication.scala:140)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543)
	at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:140)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:250)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:241)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.SocketTimeoutException: timeout
	at okio.Okio$4.newTimeoutException(Okio.java:230)
	at okio.AsyncTimeout.exit(AsyncTimeout.java:285)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211)
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
	at okhttp3.RealCall.execute(RealCall.java:69)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:226)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:769)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:356)
	... 16 more
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketInputStream.read(SocketInputStream.java:204)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at okio.Okio$2.read(Okio.java:139)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
	... 43 more

The mutating webhook seems to be there:
➜ kubectl get MutatingWebhookConfiguration
NAME                                          CREATED AT
spark-operator-sparkoperator-webhook-config   2019-07-02T07:24:40Z

Same aplies to the service:
➜ kgs -n spark
NAME                     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
spark-operator-webhook   ClusterIP   10.249.0.99   <none>        443/TCP   67m

Anyone has an idea what could be the issue?

Same problem on GKE for me, works fine on our bare metal Kubernetes though.

Interesting. Will try to reproduce it when I get back to work next week.

Can reproduce it using spark-pi example:
apiVersion: "sparkoperator.k8s.io/v1beta1"
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: spark-operator
spec:
  type: Scala
  mode: cluster
  image: "gcr.io/spark-operator/spark:v2.4.0"
  imagePullPolicy: Always
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar"
  sparkVersion: "2.4.0"
  restartPolicy:
    type: Never
  volumes:
  - name: "test-volume"
    hostPath:
      path: "/tmp"
      type: Directory
  driver:
    cores: 0.1
    coreLimit: "200m"
    memory: "512m"
    labels:
      version: 2.4.0
    serviceAccount: spark
    volumeMounts:
    - name: "test-volume"
      mountPath: "/tmp"
  executor:
    cores: 2
    instances: 3
    memory: "512m"
    labels:
      version: 2.4.0
    volumeMounts:
    - name: "test-volume"
      mountPath: "/tmp"

@doctapp did you also install the operator using the Helm Chart?

@liyinan926 yes used helm with all instructions mentioned.

@doctapp have you seen the following error in your operator logs?
failed to run spark-submit for SparkApplication default/spark-pi: failed to run spark-submit for SparkApplication default/spark-pi: Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.16.0.1/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "spark-pi-driver" is forbidden: error looking up service account default/spark: serviceaccount "spark" not found.

@liyinan926 I checked the logs and I only get the SocketTimeout exceptions. I've tried the following steps and I still get the same error:
Command used to install Spark operator
helm install incubator/sparkoperator --name spark-op --set enable-metrics=true --set enableWebhook=true

Job executed
apiVersion: "sparkoperator.k8s.io/v1beta1"
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: "gcr.io/spark-operator/spark:v2.4.0"
  imagePullPolicy: Always
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar"
  sparkVersion: "2.4.0"
  restartPolicy:
    type: Never
  volumes:
    - name: config-vol
      configMap:
        name: dummy-cm
  driver:
    cores: 0.1
    coreLimit: "200m"
    memory: "512m"
    labels:
      version: 2.4.0
    serviceAccount: spark-op-spark
    volumeMounts:
      - name: config-vol
        mountPath: /opt/spark/mycm
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 2.4.0
    volumeMounts:
      - name: config-vol
        mountPath: /opt/spark/mycm

Result (logs from spark operator)
} map[] 0 1}] to [{  2019-08-12 09:55:11 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0    } {FAILED failed to run spark-submit for SparkApplication default/spark-pi: Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create]  for kind: [Pod]  with name: [null]  in namespace: [default]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:363)
	at org.apache.spark.deploy.k8s.submit.Client$$anonfun$run$2.apply(KubernetesClientApplication.scala:141)
	at org.apache.spark.deploy.k8s.submit.Client$$anonfun$run$2.apply(KubernetesClientApplication.scala:140)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543)
	at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:140)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:250)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:241)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.SocketTimeoutException: timeout
	at okio.Okio$4.newTimeoutException(Okio.java:230)
	at okio.AsyncTimeout.exit(AsyncTimeout.java:285)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211)
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
	at okhttp3.RealCall.execute(RealCall.java:69)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:226)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:769)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:356)
	... 16 more
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketInputStream.read(SocketInputStream.java:204)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at okio.Okio$2.read(Okio.java:139)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
	... 43 more

Everything seems to be working fine when I skip the mutable admission webhook. Could it be the case that the Kubernetes API is closed off? Or do we need to enable additional features on the Kubernetes cluster on GCP? For instance something like Kubernetes alpha features or Binary Authorization?

@zwennesm no, you shouldn't need any special setup to make it work.

@liyinan926 It does seem to create a service account in the default namespace.
NAME                     SECRETS   AGE
default                  1         271d
spark-op-spark           1         117s
spark-op-sparkoperator   1         117s

~ at ☸  bijenkorf-dev
➜ k describe sa spark-op-spark
Name:                spark-op-spark
Namespace:           default
Labels:              app.kubernetes.io/instance=spark-op
                     app.kubernetes.io/managed-by=Tiller
                     app.kubernetes.io/name=sparkoperator
                     helm.sh/chart=sparkoperator-0.2.4
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   spark-op-spark-token-xswqk
Tokens:              spark-op-spark-token-xswqk
Events:              <none>

In my initial issue report I did use the sparkJobNamespace parameter, but with that settings I am getting the exact same error. Would it be worth a shot to try and install the Spark operator without the use of the Helm templates?

Yes, I think it worths trying out installation through the manifest in the operator repo.

@doctapp have you seen the following error in your operator logs?
failed to run spark-submit for SparkApplication default/spark-pi: failed to run spark-submit for SparkApplication default/spark-pi: Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.16.0.1/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "spark-pi-driver" is forbidden: error looking up service account default/spark: serviceaccount "spark" not found.

I saw serviceAccount was set to spark in the test SparkApplication you used. Can you check if that service account exists in namespace spark-operator, which is the namespace of your SparkApplication.
BTW: I could not reproduce the issue you ran into. I was able to mount the test volume using the latest Helm version on GKE 1.12.8-gke.10.

The spark service account is in the spark-operator namespace. Here are the commands I used to create my cluster and deploy the spark operator:
gcloud beta container --project xxx clusters create gke-spark-test --zone us-xxx --no-enable-basic-auth --cluster-version "1.12.8-gke.10" --machine-type n1-standard-4 --image-type "COS" --num-nodes 3 --enable-stackdriver-kubernetes --enable-private-nodes --enable-private-endpoint --master-ipv4-cidr x.x.x.x/x --enable-ip-alias --network "xxx" --subnetwork "xxx" --enable-master-authorized-networks --master-authorized-networks x.x.x.x/x --addons KubernetesDashboard --maintenance-window 03:00 --enable-autoupgrade --enable-autorepair
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-role --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
kubectl create namespace spark-operator
kubectl create serviceaccount spark -n spark-operator
kubectl create clusterrolebinding spark-role --clusterrole=cluster-admin --serviceaccount=spark-operator:spark --namespace=spark-operator
helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
helm install incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=spark-operator --set imagePullPolicy=Always --set enableWebhook=true

@doctapp you don't need to manually create a service account and a role binding. The Helm installation will create a service account and role binding in the namespace spark-operator for you. In your case, the service account will be named spark-operator-spark.

@doctapp and @zwennesm if you could try installing the operator using the manifest following the instruction below, it would help determining if the issue you ran into is with the Helm chart:
# cd spark-on-k8s-operator
# kubectl apply -f manifest/spark-operator-rbac.yaml
# kubectl apply -f manifest/spark-operator-with-webhook.yaml
# kubectl apply -f manifest/spark-rbac.yaml

This will setup up RBAC and install the operator into the namespace spark-operator. It will also create a service account named spark in the default namespace for your Spark applications. To test the setup, run the example application:
kubectl apply -f examples/spark-pi.yaml

@liyinan926 Following the exact steps you mentioned I still get the same error as before:
} map[] 0 1}] to [{  2019-08-12 19:45:47 +0000 UTC 0001-01-01 00:00:00 +0000 UTC { 0    } {FAILED failed to run spark-submit for SparkApplication default/spark-pi: Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [create]  for kind: [Pod]  with name: [null]  in namespace: [default]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:363)
	at org.apache.spark.deploy.k8s.submit.Client$$anonfun$run$2.apply(KubernetesClientApplication.scala:141)
	at org.apache.spark.deploy.k8s.submit.Client$$anonfun$run$2.apply(KubernetesClientApplication.scala:140)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543)
	at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:140)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:250)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$5.apply(KubernetesClientApplication.scala:241)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2543)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241)
	at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.SocketTimeoutException: timeout
	at okio.Okio$4.newTimeoutException(Okio.java:230)
	at okio.AsyncTimeout.exit(AsyncTimeout.java:285)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211)
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
	at okhttp3.RealCall.execute(RealCall.java:69)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:226)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:769)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:356)
	... 16 more
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketInputStream.read(SocketInputStream.java:204)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at okio.Okio$2.read(Okio.java:139)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
	... 43 more
} map[] 0 1}]
I0812 19:45:47.219246       9 controller.go:253] Ending processing key: "default/spark-pi"

Now could it be that one of the additional GKE settings are interfering with the mutating admission webhooks feature? We have the following settings enabled in our cluster: VPC-native (alias IP), Private cluster. And additionally we disabled the HTTP load balancing feature.

It could be because of the cluster settings. One more ask, could you create a temporary standard cluster and try it again? If it works, it's clearly that certain cluster settings will interfere with the mutating webhook.

Found this https://github.com/kubernetes/autoscaler/issues/1820.

@doctapp you don't need to manually create a service account and a role binding. The Helm installation will create a service account and role binding in the namespace spark-operator for you. In your case, the service account will be named spark-operator-spark. Let me know if using the service account spark-operator-spark works for you.

Tried that and am now getting: "error looking up service account spark-operator/spark-operator-spark: serviceaccount "spark-operator-spark" not found"

@doctapp interesting. Can you list all the service accounts in namespace spark-operator?

@zwennesm @doctapp I think you both ran into the same issue with webhooks on a private cluster.  The default port for the webhook service is 8080, which seems not allowed for private clusters according to https://github.com/kubernetes/autoscaler/issues/1820. We do allow changing the port through the parameter webhookPort . Can you try doing the Helm installation with the following flag?
--set webhookPort=443

@liyinan926 Changing the port to 443 using the attribute you suggested worked! I tested it using the following Helm command:
helm install incubator/sparkoperator --name spark-operator --namespace spark --set sparkJobNamespace=spark --set enable-metrics=true --set enableWebhook=true --set webhookPort=443

@doctapp this will create a service account in the spark namespace named spark-operator-spark:
➜ k get sa -n spark
NAME                           SECRETS   AGE
default                        1         89m
spark-operator-spark           1         89m
spark-operator-sparkoperator   1         89m

FYI: To make it work you have to change the namespace of the actual SparkApplication to run in the same namespace where you configured the service-account and the sparkJobNamespace to be. The SparkApplication below works for me.
apiVersion: "sparkoperator.k8s.io/v1beta1"
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: spark
spec:
  type: Scala
  mode: cluster
  image: "gcr.io/spark-operator/spark:v2.4.0"
  imagePullPolicy: Always
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar"
  sparkVersion: "2.4.0"
  restartPolicy:
    type: Never
  volumes:
    - name: "test-volume"
      hostPath:
        path: "/tmp"
        type: Directory
  driver:
    cores: 0.1
    coreLimit: "200m"
    memory: "512m"
    labels:
      version: 2.4.0
    serviceAccount: spark-operator-spark
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 2.4.0
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"

Thanks a lot for your patience and help @liyinan926. You really helped me out. Should I create a PR in the GCP docs to explain this issue?

@zwennesm Had to use --name spark-operator otherwise it gets a random name which I didn't check. Many thanks :)
@liyinan926 I too confirm that --set webhookPort=443 works. Thanks a lot for finding this :)

Cool! @zwennesm I was also thinking about updating the doc. It would be great if you can create a PR. Thanks!

Closing this and looking forward to the PR for doc enhancement. Thanks!

I still have the problem when I disable the mutation webhook. Used to set a larger memory for the operator and solved it. Now it comes again.
spark-operator:v2.4.0

It is solved by set export KUBERNETES_REQUEST_TIMEOUT=100000

Hello @liyinan926 & All,
I ran into the same issue and wanted to perform a clean reinstallation with the solution provided (setting the webhook port), what do I need to do to achieve this? What are the uninstallation steps of the operator to do before executing the reinstallation? Thanks in advance.

It is solved by export KUBERNETES_REQUEST_TIMEOUT=100000
which is the environment variable for https://github.com/fabric8io/kubernetes-client#configuring-the-client

May I know how to set KUBERNETES_REQUEST_TIMEOUT env variable?

It is solved by export KUBERNETES_REQUEST_TIMEOUT=100000
which is the environment variable for https://github.com/fabric8io/kubernetes-client#configuring-the-client

May I know how to set KUBERNETES_REQUEST_TIMEOUT env variable?

I came across the same issue. And setting the webhook port works for me.
--set webhookPort=443

FYI: webhookPort is changed to webhook.port
for example, --set webhook.port=443

authors: doctapp fisache gaocegege kehao95 liyinan926 marwanfahmi tmclouisluk zwennesm


text:
Mark metav1.Time fields as being nullable

@akhurana001

looks good. Thanks for getting to this.  I will rebase the other PR once this is checked this.

@akhurana001 thanks for the quick review! Actually I just realized nullable is not supported before 1.14. I made a comment on your PR, and I think we should just remove validation for status all together for now.


text:
Update the rl-pong example for tf 1.10

There was a compatibility issue from TensorFlow 1.8 to 1.9, where the variable reuse behavior was changed when building a model with Keras.
Related issue: https://github.com/GoogleCloudPlatform/tensorflow-without-a-phd/issues/20
(Possibly related issue: https://github.com/GoogleCloudPlatform/tensorflow-without-a-phd/issues/11)

Thanks you for the fix. Did you test by training again and are you sure that this new model does not require a new "demo checkpoint" in the "demo-checkpoint" folder ?

retrained and replaced demo-checkpoint.

Thanks, merging

authors: dizcology martin-gorner


text:
Enhanced logging

The feature branch is logging, but it's more accurately an observability/monitoring branch where we will:

Add Spring Cloud GCP dependency, logback config to use stackdriver, and skaffold config to set active profiles appropriately
Add Cloud trace for distributed request tracing

Merged into #7 for simplicity


text:
fix(terraform): fix terraform module versions

Description

Fix Terraform module versions
Add default_language_version to .pre-commit-config.yaml

How has this been tested?
Tested with terraform apply locally.
Checklist

[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have successfully run the E2E tests, and have included the links to the pipeline runs below
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] I have updated any relevant documentation to reflect my changes
[ ] I have assigned a reviewer and messaged them

Pipeline run links:

It's a fair point!
These docs were (mostly) autogenerated using terraform-docs, which works great for reusable Terraform modules that are published to the Terraform Registry.
Since I don't think we're likely to publish the terraform directory as a module to the Terraform Registry, let's remove this stuff from the README (everything from <!-- BEGIN_TF_DOCS --> below can be cut). It will make it neater and easier to maintain.
Thanks!

Done! I removed the auto-generated terraform-docs from README.

/lgtm
/merge

@Linchin I don't think we have any of this automation set up. Is this something you can help us with?

@Linchin I don't think we have any of this automation set up. Is this something you can help us with?

Yes, I merged it as I just became an admin for this repo. Let me see how to do the / lgtm and / approve thing.

authors: Linchin browningjp-datatonic kawo123


text:
How to specifying mvn project version as docker image version

mvn project version is increase

you can use ${project.version} in to/image

e.g.
<to>
  <image>${docker.image.prefix}/${project.artifactId}:${project.version}</image>
</to>

right?

thanks.

Sounds like it's solved, feel free to reopen if not.

authors: TadCordle ssoerensen zeusbaba zong400

repo: GoogleContainerTools/jib


text:
Skip validating non-central root reconciler Pods in validateMultiRepoPods

Some tests failed with a namespace reconciler Pod not found error, e.g. https://oss.gprow.dev/view/gs/oss-prow-build-kpt-config-sync/logs/multi-repo-7-standard-stable/1583109762550075392. The root cause is that the ValidateMultiRepoDeployments function validates the Pods after the Deployment objects are ready. When validating the Pods, it checks all Pods in the c-m-s namespace, but when validating the Deployments, it only checks the default root reconciler and other controllers. It is possible that the namespace reconciler Deployment is not available yet, which results in validation error for the corresponding Pod.
This commit adds the namespace reconciler Deployment to the validation list.
It also adds the error to the log when the git-server fails to be installed:
https://oss.gprow.dev/view/gs/oss-prow-build-kpt-config-sync/logs/stress-test/1581530245859643392.

This PR should fix the pods "ns-reconciler-bookinfo-xxxx" not found failure in the following test:

TestOverrideGitSyncDepthV1Beta1
TestCACertSecretRefV1Alpha1
TestCACertSecretWatch
TestDeleteRepoSync_Centralized_AndRepoSyncV1Alpha1
TestSwitchFromGitToOci
...

/lgtm

/approve

authors: nan-yu sdowell tiffanny29631

repo: GoogleContainerTools/kpt-config-sync


text:
Design system website: Contribute page

Acceptance Criteria:
-Add the following page to the design system website

Note: The "Contribute" page should sit below the "QA testing" page in the left nav.
The existing "Contribute" page in the support section should be removed as part of this work.
Copy the existing "Contribute" page over and then edit

This is done here: https://github.com/GovAlta/ui-components-docs/pull/239

repo: GovAlta/ui-components