text:
interceptor missing (not intercepting) multiple requests

Steps to reproduce
Tell us about your environment:

Puppeteer version: 3.3.0
Platform / OS version: Win10/64bit
URLs (if applicable): kbb.com
Node.js version: 10.16.3

What steps will reproduce the problem?
See code below. I expected that ALL requests will pass through the interceptor, but many requests that appear in DevTools are never acknowledged, thus no opportunity to "catch.". #4329 is similar, but was resolved via bugfix in 1.15, however I'm on 3.3.
Please include code that reproduces the issue.
(code block below)
What is the expected result?
I expect that the call to https://accounts.us1.gigya.com/accounts.login to be intercepted.
What happens instead?
It is ignored in the interception code block page.on('request', ...  The request goes through without the opportunity to intercept. For debugging purposes, the interception block writes all URL's to console after we set clicked=true, but this URL doesn't appear there.
(the call to https://accounts.us1.gigya.com/socialize.getSavedResponse also fails to be recognized by the interception code block)
code:
const puppeteer = require('puppeteer');

const getPuppeteer = async () => {
  let browser = await puppeteer.launch({
    timeout: 120000,
    headless: false,
    devtools: true,
    //slowMo: 100,
  });
  return new Promise((resolve, reject) => {
    resolve(browser)
  });
}

async function kbb() {

  let exclusions = /\.gif|\.png|\.jpg|\.ttf|facebook|doubleclick|nr-data|foresee|demdex|pinterest|yimg|clarivoy|marinsm|mavenaudiences|go-mpulse|zemanta|esm1|adnxs|adsrvr|rockerbox|arrivalist|googleads|googlesyndication|google-analytics|googletagmanager/;

  let clicked = false;
  let browser = await getPuppeteer(); //cloud || false);
  try {
    const page = await browser.newPage();

    // this is our interceptor, where we capture the browser's POST
    page.on('request', async(request) => {
      if (clicked) console.log(request.url() + "\n") // display ALL request URLs after the "Sign In" button was clicked
      if (request.url().search(/login/g) > 0) {
        // if (request.method == "POST") {
        // if (request.url().includes("accounts.login")) {
        console.log("yes!")
        console.log(request.url());
        console.log(request.method());
        let body = request.postData();
        console.log(body);
        request.abort();
      } else if (request.url().search(exclusions) > 0) {
        request.abort();
      } else {
        request.continue();
      }
    });

    await page.setRequestInterception(true);
    await page.goto("https://www.kbb.com/mykbb/");
    await page.waitForSelector(".mykbb-auth");

    await page.type("#loginEmail", 'githubuser@gmail.com');
    await page.type("#loginPassword", 'ilovecars');
    console.log("clicking...");
    clicked = true; // set to true just before clicking... for debugging we can set the interceptor to catch everything when true
    await page.click("#myKbbSignin");

  } catch(err) {
    console.log(err);
  } finally {
    if (browser) {
      // browser.close();
      // browser.disconnect();
    }
  }
}

(async () => {
  let my_kbb = await kbb()
  console.log("done!")
})();

Similar here
"puppeteer": "^5.3.1"
Requests with type of 'document' are caught - others aren't.

Same issue here, pptr 5.4.1 - I can send the site if a developer reaches out to me. Chromium debugger shows about 20 requests, some are intercepted (and aborted by me) but others are not intercepted at all. I'm not using the userDataDir option.

Exact same issue as well, puppeteer 5.5.0. Seems to miss specific types of types. like .m3u8 or .ts (streaming files). #5130

If anybody is still having problems, using await page.setBypassServiceWorker(true) solves the problem!

I'm having the issue as well, the solution above didn't work.
You need to go lower level.
here's what worked:
    const cdp = await page.target().createCDPSession();
    await cdp.send('Network.enable');

    cdp.on('Network.responseReceived', (params) => {
       console.log(params.response.url)
    });

     // Get the response body
    const response = await client.send('Network.getResponseBody', {
      requestId: params.requestId,
    });

    // Output the response body
    console.log(response.body); // This will be either the content or base64-encoded data depending on the response type
    // If the response is base64-encoded, you need to convert it
    if (response.base64Encoded) {
      const buffer = Buffer.from(response.body, 'base64');
      console.log('Decoded response:', buffer.toString());
    }

If your browser already had cached the resource, puppeteer will not get to intercept it since the resource will directly be served from the cache.
To disable the cache, use the below
await page.setCacheEnabled(false)


text:
(PUP-9747) Reenable resource api tests, test against hotfix branch

Duplicate of #1032 since travis ci wasn't picking that one up

CLA signed by all contributors.

Closing in favor of BOLT-1385

source: gharchive/pull-request

authors: DavidS adreyer puppetcla

repo: puppetlabs/bolt

license_type: permissive

license_source: github-api


text:
(maint) Pin yard gem due to introduction of bug in puppet-strings

This commit pins the yard gem to avoid unwanted behavior in puppet-strings rake task to generate plan function docs. For example with 0.9.20 out::message becomes :'out::message'.

CLA signed by all contributors.

authors: donoghuc puppetcla


text:
(FACT-1587) split out or adde command line override tests

I split out one test and created the other to verify command line override of the
configuratino file. I also updated some of the old test since I was there.

Tested and passing on:
arista4-32a
ciscoxr-64a%7Bhypervisor=vmpooler%7D
cumulus25-64a
debian8-64a
fedora25-64a
osx1012-64a
redhat7-64a
sles12-64a
solaris11-64a
ubuntu1604-64a
windows10ent-64a
windows2016-64a

CLA signed by all contributors.

authors: ericldelaney puppetcla

repo: puppetlabs/facter

license_source: bigquery


text:
(SERVER-1150) Add support for custom oid mappings

Add support for Puppet's custom trusted oid mapping feature.
This is a cherry-pick of the commit in https://github.com/puppetlabs/puppet-server/pull/996

@fpringvaldsen the puppet agent version will need updating to a version that includes in OID work as well

I've updated the submodule commit and fixed the failing test

@haus

the puppet agent version will need updating to a version that includes in OID work as well

Do you mean for the CI Pipeline to pass? Do you know where I can set this?

acceptance/lib/helper.rb has a sha listed. the sha will need to be an agent version that includes the puppet changes. you can see #1007 for an example.

And yea, I meant for acceptance tests to pass.

submodule SHA looks good, I think this is ready except for the change @haus mentioned

@KevinCorcoran @haus I've updated the puppet-agent dependency, hopefully that should make CI green

👍

:+1: will merge once travis goes green

authors: KevinCorcoran fpringvaldsen haus

repo: puppetlabs/puppet-server


text:
(MAINT) Bump project.clj version to 2.2.2-master-SNAPSHOT

This commit bumps the Puppet Server version in the project.clj up to
2.2.2-master-SNAPSHOT given that we've released 2.2.1 already.

Any reason this can't be 2.3.0-SNAPSHOT ?

I don't think we know for sure yet that the next release we do off of this branch will be a Y bump.  We've tried to be conservative about that in the past anyway - only bump to the next Z.

I'm +1 on this ending up as 2.2.2-master-SNAPSHOT, but it seems like if it currently says 2.2.0 then we are delinquent on some merge-ups (which is probably my fault).  In any case I'd have no big problem with merging this now as long as we're careful about not screwing it up when the merge-ups come in.

@cprice404 I think we had already merged up code from stable recently.  Just did a separate merge up PR to catch everything else that was in stable but not master yet - just docs.  That PR is #817.  Figured it made sense to have the version bump and merge up be on separate PRs.

:+1:


text:
(maint) Remove comment referencing trac

The comment at the start of lib/puppet/reference/providers.rb was added in https://github.com/puppetlabs/puppet/commit/7835d2927ef360216612f1a782e88a6606a79d6b in 2007.
As we no longer use trac, this commit removes that comment.

But ... I loved that comment so much. Sniff.

Also: since the PR branch was on upstream, I went ahead and deleted that branch after merge.

authors: haus kylog

repo: puppetlabs/puppet


text:
(maint) Update file source docs

Document explicit checksum_value with source, X-Checksum-* headers,
userinfo component and HTTPS servers with ssl_trust_store setting.

CLA signed by all contributors.

Thanks for the feedback @clairecadman! I'll update this PR later today

I think I got all of your suggestions accept for HTTP Basic Authentication as it's the name of a specific type of authentication: https://en.wikipedia.org/wiki/Basic_access_authentication

authors: joshcooper puppetcla


text:
(PDB-2251) Document unchanged resources for PE 2015.3.0 (Ankeny)

This commit documents unchanged resources information for PE Ankeny in
the FOSS documentation.

Refer to this link for build results (access rights to CI server needed):
http://jenkins-enterprise.delivery.puppetlabs.net/job/enterprise_puppetdb_init-multijob_githubpr-stable/96/
Test PASSed.

@kbarber updated per our discussion!

Refer to this link for build results (access rights to CI server needed):
http://jenkins-enterprise.delivery.puppetlabs.net/job/enterprise_puppetdb_init-multijob_githubpr-stable/98/
Test FAILed.

@kbarber @wkalt Updated with your guys' input and the input from docs!

Refer to this link for build results (access rights to CI server needed):
http://jenkins-enterprise.delivery.puppetlabs.net/job/enterprise_puppetdb_init-multijob_githubpr-stable/100/
Test FAILed.

authors: ajroetker puppetlabs-jenkins

repo: puppetlabs/puppetdb


text:
Add a facter fact for dist-upgrade

This commit adds a facter fact for dist-upgrade, along with
associated facter facts.
(Builds upon PR https://github.com/puppetlabs/puppetlabs-apt/pull/634 - adds a new set of facts to maintain compatibility)

i took a peek at tests. may have something to do with scoping in RSpec. while i was testing apt_dist_has_updates_spec.rb, the fact was returning nil, not even false for the last example but putting File.expects(:executable?).with('/usr/bin/apt-get').returns true in the actual example (instead of the before block), it seemed to help a little.

@eputnam I can't replicate this, if I add the expect outside of the block then it returns nil, the opposite of what you are seeing. Travis has gone green too. Let's discuss this later.

Facts in readme updated, thanks

Thanks @eputnam

authors: eputnam willmeek

repo: puppetlabs/puppetlabs-apt


text:
渲染echarts，要么空白，要么报错

描述问题 (Describe the problem)
我已 很详细很详细 阅读文档中关于 echarts 使用的部分，main.ts中的代码如下：
import App from "./App.vue";
import router from "./router";
import { setupStore } from "@/store";
import ElementPlus from "element-plus";
import { getServerConfig } from "./config";
import { createApp, Directive } from "vue";
import { MotionPlugin } from "@vueuse/motion";
import { useEcharts } from "@/plugins/echarts";
import { injectResponsiveStorage } from "@/utils/responsive";
import Table from "@pureadmin/table";
import * as echarts from "echarts";

// 引入重置样式
import "./style/reset.scss";
// 导入公共样式
import "./style/index.scss";
// 一定要在main.ts中导入tailwind.css，防止vite每次hmr都会请求src/style/index.scss整体css文件导致热更新慢的问题
import "./style/tailwind.css";
import "element-plus/dist/index.css";
// 导入字体图标
import "./assets/iconfont/iconfont.js";
import "./assets/iconfont/iconfont.css";
// 导入unocss
import "uno.css";

const app = createApp(App);

// 全局注册echarts
app.config.globalProperties.$echarts = echarts;

// 自定义指令
import * as directives from "@/directives";
Object.keys(directives).forEach(key => {
  app.directive(key, (directives as { [key: string]: Directive })[key]);
});

// 全局注册`@iconify/vue`图标库
import {
  IconifyIconOffline,
  IconifyIconOnline,
  FontIcon
} from "./components/ReIcon";
app.component("IconifyIconOffline", IconifyIconOffline);
app.component("IconifyIconOnline", IconifyIconOnline);
app.component("FontIcon", FontIcon);

// 全局注册按钮级别权限组件
import { Auth } from "@/components/ReAuth";
app.component("Auth", Auth);

getServerConfig(app).then(async config => {
  app.use(router);
  await router.isReady();
  injectResponsiveStorage(app, config);
  setupStore(app);
  app.use(MotionPlugin).use(ElementPlus).use(useEcharts).use(Table);
  app.mount("#app");
});

现在遇见的问题是：
function onInit() {
  // 如果直接使用，页面上不会渲染出来
  //initEcharts();

  // 如果通过接口请求后再使用，则会报错：Uncaught (in promise) TypeError: Cannot destructure property 'appContext' of '(0 , c.getCurrentInstance)(...)' as it is null.
  // 不仅是异步请求之后，使用nextTick 也会报同样的错误
  DataApi.getData(toRaw(form)).then(data => {
    console.log(data);
    initEcharts();
  });
}

function initEcharts() {
  const echartRef = ref();
  const { setOptions } = useECharts(echartRef);
  setOptions({
    tooltip: {
      trigger: "item"
    },
    legend: {
      top: "5%",
      left: "center"
    },
    series: [
      {
        name: "统计",
        type: "pie",
        radius: ["40%", "70%"],
        avoidLabelOverlap: false,
        itemStyle: {
          borderRadius: 10,
          borderColor: "#fff",
          borderWidth: 2
        },
        label: {
          show: false,
          position: "center"
        },
        emphasis: {
          label: {
            show: true,
            fontSize: 40,
            fontWeight: "bold"
          }
        },
        labelLine: {
          show: false
        },
        data: [
          { value: 100, name: "周一" },
          { value: 200, name: "周二" },
          { value: 300, name: "周三" },
          { value: 400, name: "周四" },
          { value: 500, name: "周五" }
        ]
      }
    ]
  });
}

onMounted(() => {
  onInit();
});

 <div ref="echartRef" class="w-full h-[100px]" />

如何复现该问题 (How to reproduce the problem)
同上
我真的已经很测试很多次了，使用原生的echarts没问题，所以请您仔细看看，谢谢。
操作系统和浏览器信息 (Operating system and browser information)
No response
验证 (Verify)

[X] 是否仔细阅读过 文档 (Have you read documentation carefully)
[X] 检查是否存在相同或类似的问题 issues (Check for the same or similar issues)

目前只解决bug
这属于你的业务问题，自行参考demo

source: gharchive/issue

repo: pure-admin/vue-pure-admin

license: MIT


text:
adds a try function

turns exceptions into Eithers, adresses #15

👍

Just as a question. Do we prefer unicode arrows, forall and double colon in libraries?

I've been avoiding them, as with the core libraries being one place newcomers are likely to encounter PS code, I don't want to scare them off with "crazy symbols" or something.

Sounds good, I can empathize with that.

authors: garyb kRITZCREEK paf31

repo: purescript/purescript-exceptions


text:
🛑 Social Connector Backend Core is down

In 1b4f11d, Social Connector Backend Core (https://devcore.socialconnector.io/api/health) was down:

HTTP code: 502
Response time: 621 ms

Resolved: Social Connector Backend Core is back up in 8032484.

repo: puresec-ng/socialconnector-uptime


text:
[Improvement]: Add Deployment Live Project Link in About Section

GSSOC'23
Issue Description:

Add Deployment Live Project Link in About Section.

Advantage:

Advantage of this Improvement is of User whenever visits this Repository. Easily Navigate to Live Project Demo and Find Bugs & Issues and Feature Requests Earlier.

Can you please assign me this issue? I would like to work on it.

Yes please
You can work on it @NarendraKoya999


text:
Integration in Swift Server on Linux

I have a Kitura based server running on Heroku (Linux). Setting up push notifications there has proved very challenging, because Heroku don't have HTTP/2 yet. I am tempted to use a service like Pusher instead, but that would mean that I need this project to run on Linux. Has that been tested?

This library is for the iOS / macOS component, i.e. the client that received the push notifications. To trigger push notifications using the Pusher push notifications service you’ll want to send a request to our HTTP endpoint: https://pusher.com/docs/push_notifications/reference/server_api
We don’t currently have a Swift HTTP client to publish new push notifications but if this is something that you’d be interested in then we can certainly take a look at creating one, seeing as it’s not too difficult. On that note you may well find that you’re able to easily make the necessary requests yourself using URLSession or similar.

On 29 Sep 2017, at 17:56, Drew McCormack notifications@github.com wrote:
I have a Kitura based server running on Heroku (Linux). Setting up push notifications there has proved very challenging, because Heroku don't have HTTP/2 yet. I am tempted to use a service like Pusher instead, but that would mean that I need this project to run on Linux. Has that been tested?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Ah, OK. Thought it might be capable of posting events as well. I will look into just doing http requests from the server.

authors: drewmccormack hamchapman

repo: pusher/pusher-websocket-swift

license: mit


text:
🛑 PMB is down

In 9935b05, PMB (https://pmb.unmer.ac.id/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: PMB is back up in 6fb0d5c.

repo: pusimgit/upptime


text:
Getting loads of recommendations on single entry page

Question
I am not sure I am getting this right: On my overview page template the plugin successfully detected loads of non-eager loaded for-loops. On my single entry page however I get loads of recommendations, where I am not sure these can be eager loaded at all.
e.g. I have a hero image that is exclusive to the entry I am rendering, through that dedicated template for the entry, like this (I am using ImagerX for the images):

    {% set image = entry.heroImage.one() %}
    {% set transform = craft.imagerx.transformImage(image, 'standardImage') %}

    {{ image.title }}

The plugin says I should eager load the heroImage.
From what I understand, there is no need here to load anything eagerly as it is one straight request to that asset within the entry I am in anyway.
Can you give me a hint on that, is the plugin just searching for suspicious calls to .one() or did I miss a crucial point in the whole eager loading approach?

This sounds a lot like the question that was asked in https://github.com/putyourlightson/craft-blitz-recommendations/issues/3, hopefully that'll help clarify things for you.

Version 2.1.0 improves the behaviour of hints, see https://putyourlightson.com/articles/ballroom-blitz

Perfect, thank you! Looking forward to check out the new versions.

authors: bencroker tsawitzki


text:
Can you set a unique id?

We have a trace id coming from a third party service and would like to use that one, in that request for the logs in our service that calls it. is that possible? here is an example of what we are hoping for
const rTracer = require('cls-rtracer');

const fakeEndpoint = async (req, res) => {
  const traceIdFromThirdParty = req.traceId;

  rTracer.setId(traceIdFromThirdParty);
}

Hi. requestIdFactory config param serves exactly this purpose:
const rTracer = require('cls-rtracer');

app.register(rTracer.fastifyPlugin, {
  requestIdFactory: (req) => req.traceId
});

const fakeEndpoint = async (req, res) => {
  console.log('here is my 3rd-party request id: ' + rTracer.id());
};

hey @puzpuzpuz , dumb question, this is assuming app is the server and your adding it like a middleware? this may not work for us, the server lib we use doesnt allow custom middlewares, is there any other option?

@puzpuzpuz hello, thanks for the reply, and ya we do have a web server but its a custom company made one that doesnt allow injection of middlewares (really stupid i know crys), so inthis example, we would need to set the id on an method (now that i look at it, no request at all object just a normal function) then use that trace id rest of the code base that calls the id function, if that is able to so with AsyncLocalStorage, do you have any examples? thanks !

ok ill look, thank you, ill close this but i may have question in future :) thanks!

authors: TannerS puzpuzpuz

repo: puzpuzpuz/cls-rtracer


text:
Update sbt-sonatype to 3.9.14

Updates org.xerial.sbt:sbt-sonatype from 3.9.3 to 3.9.14.
GitHub Release Notes - Release Notes
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.xerial.sbt", artifactId = "sbt-sonatype" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "org.xerial.sbt", artifactId = "sbt-sonatype" }
}]

labels: sbt-plugin-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #151.

repo: pwliwanow/fdb-pubsub


text:
Unc0ver freezes on 2/31 Exploiting kernel

My iPhone:
iPhone 6s Plus
IOS 13.5.1
I installed with AltStore with AltServer on a Windows computer.

unc0ver does not support 13.5.1
Use checkrain.

unc0ver does not support 13.5.1
Use checkrain.

What is the latest version it supports?

https://support.apple.com/en-us/HT211214
this is a link to apple website describing why 3.5.1 was released and what it fixes.  You will notice it only mentions fixing memory consumption impacted by an application to execute arbitrary code with kernel privileges.  They name Unc0ver specifically.
What this means is 13.5.1 was written solely to stop the unc0ver jailbreak.  If you have an older device, then you could possibly jailbreak using checkrain. They still don't offer windows support.  So I myself will not be exploring anything checkrain. You will have to find that information yourself.
good luck.

Ty

No problem.


text:
Unc0ver Crashing After Attempted Jailbreak

Describe the bug
After restarting, my iPhone 11 on 13.5 is not able to jailbreak. The app attempts to jailbreak, and then crashes without completing it. The app continues to crash unless rebooted
To Reproduce
Steps to reproduce the behavior:

Open Unc0ver App
Click on 'Jailbreak'
Wait
It'll get to about the 28th step and then just crash

Screenshots
Before restoring Rootfs: https://youtu.be/bqbA07wN7LM
After restoring Rootfs: https://youtu.be/6WUY6f-MEc8
Device (please complete the following information):

iOS Version: 13.5
iOS Device: iPhone 11
unc0ver Version: Tested on 5.0.1, 5.2.0, 5.2.1

Place an "x" between the brackets if true:

[ ] this is a bug others will be able to reproduce
[x] this issue is present with all tweaks uninstalled(except for default packages) or disabled
[x] this issue is present after a rootfs restore
[x] this issue is present on the latest version of unc0ver

Logs
If applicable, add logs or error messages here.
Present in videos linked

go to settings, general, reset all settings.  When you go to jailbreak, restore root FS first.  then reboot and try again.

You might also want to use a different cord when installing unc0ver.

go to settings, general, reset all settings. When you go to jailbreak, restore root FS first. then reboot and try again.

Restoring settings didn't work unfortunately

You might also want to use a different cord when installing unc0ver.

What do you mean a cord? I went to the website and used the "Open in AltStore" option and for the older releases, I had them in my Files

If you look closely in that video at the 25 second mark.  you can see that you have an unauthenticated package "substituted"   ..   Also you left Gump installed before you did all of this.  That might be whats causing all this.
You need a high level programmer to help you with this.

If you look closely in that video at the 25 second mark. you can see that you have an unauthenticated package "substituted" .. Also you left Gump installed before you did all of this. That might be whats causing all this.
You need a high level programmer to help you with this.

I see, thank you so much for this information. Theoretically, a factory reset should fix this right?

totally. But you will lose your jailbreak.  I think if you just run it normally (unjailbroken) until more information can be found about this.  I'm very confident it can easily be fixed.

just curious.  have you tried jailbreaking with "load tweaks" unchecked?

just curious. have you tried jailbreaking with "load tweaks" unchecked?

Just tried it now and the same thing happened

You tried redownloading unc0ver yet?

You tried redownloading unc0ver yet?

yeah, I've tried redownloading as well as using older versions but with no luck

totally. But you will lose your jailbreak. I think if you just run it normally (unjailbroken) until more information can be found about this. I'm very confident it can easily be fixed.

Update on this. I tried Factory Resetting, and even that didn't work

When you say factory reset,  did you just click reset everything?  or did you actually do a full upgrade with restore?

When you say factory reset,  did you just click reset everything?  or did you actually do a full upgrade with restore?

I actually ended up fixing the issue. I did a factory reset and it continued to do it, but after continuously attempting to jailbreak it wouldn't work. For some reason, it worked almost instantly with the wifi off and Siri off as well

@VMT-Batman thanks, the trick worked for me as well

authors: Hamlock-maneuverr VMT-Batman bagbyte


text:
Make sudo behave like sudo

It would be great if the dojo's implementation of sudo supported some of real sudo's flags.
The minimum ask here is -u to specify a user.  I know other dojo developers have had the desire to drop privileges in .init and execute something as non-root.  This currently can be done with runuser or su -c, but sudo is a reasonable first program to think of to perform this task.

In this file. I'll probably add it this weekend

Added --user/-u support in e0e66f6cc9d9cdac3daeef2effb5ccde96588f4b.
If we want to support more of the features, we can implement them when they are requested in a new issue.

In this file. I'll probably add it this weekend

If you want to investigate something this weekend, I'd appreciate your insight on https://github.com/pwncollege/dojo/issues/563!

authors: ConnorNelson robwaz supercoolspy

repo: pwncollege/dojo


text:
Improve the Octo pr list command

Based on #546 with original work done by @alexkalderimis
Addressed review feedback by @pwntester

Add --paginate to API call
Validate and document new configuration field pull_requests.limit
Handle nil pull_request author, author.id

Handle nil pull_request author.username
Change default pull_requests.limit from 50 to 100

@legobeat thanks so much for this!

This doesn't work for me still. I commented out the "--paginate" as it isn't a flag in the command used. I prefer the previous entry_maker a bit more. Plus with the colors & symbols here: https://github.com/pwntester/octo.nvim/pull/637

Also noting that this doesn't impact the Octo pr search command. Is that as intended?

I think much of the original requirements can be done with the current Octo pr search and being combined with the advanced search functionality: https://docs.github.com/en/search-github/searching-on-github/searching-issues-and-pull-requests
If the goal is to have it in the fuzzy finder and move away from GitHub search then that is another item.
What do people think?

I think much of the original requirements can be done with the current Octo pr search and being combined with the advanced search functionality: https://docs.github.com/en/search-github/searching-on-github/searching-issues-and-pull-requests

one slight problem with Octo pr search is that one can't merge/checkout PRs right in telescope with picker bindings (these bindings work for me only in Octo pr list)

If the goal is to have it in the fuzzy finder and move away from GitHub search then that is another item.
What do people think?

IMHO it's nice to have branch/author in fuzzy finder

IMHO it's nice to have branch/author in fuzzy finder

I guess I find the picker a bit cumbersome in the current form. How about adding the formatting of the user and the branch as done in the PR / issue details in order to provide some separation of the row information.

one slight problem with Octo pr search is that one can't merge/checkout PRs right in telescope with picker bindings (these bindings work for me only in Octo pr list)

Adding those bindings could be a good addition and easier solution for this problem.

one slight problem with Octo pr search is that one can't merge/checkout PRs right in telescope with picker bindings (these bindings work for me only in Octo pr list)

Adding those bindings could be a good addition and easier solution for this problem.
That would be done here: https://github.com/wd60622/octo.nvim/blob/88d9fe8e90fa3f5fdeedf4d8d7c1b1abf27f825f/lua/octo/pickers/telescope/provider.lua#L558-L559
in the same manner as:
https://github.com/wd60622/octo.nvim/blob/88d9fe8e90fa3f5fdeedf4d8d7c1b1abf27f825f/lua/octo/pickers/telescope/provider.lua#L341-L344
But ideally, the user would be able to configure these a bit more.

do you mind having a look at https://github.com/pwntester/octo.nvim/pull/647?

Apologies for dropping a half-baked version without timely follow-up!

This doesn't work for me still. I commented out the "--paginate" as it isn't a flag in the command used.

Thanks. I have rebased on master now, fixed that oversight and done some manual testing now.

I also prefer the previous entry_maker a bit more. Plus with the colors & symbols here: https://github.com/pwntester/octo.nvim/pull/637

Nice!

Also noting that this doesn't impact the Octo pr search command. Is that as intended?

There are some tradeoffs with each approach (see description on #546 for some motivations). One one hand I'm not sure too what extent it makes sense to have separate search/list commands with differing behavior and implementation but on the other it is convenient to use it as differentiator between calling graphql or not, as either can be desirable and it would be a chore to have to configure desired default for each repo... So maybe a single unified implementation which can be configured, but still two differing commands with differing defaults is a better approach...?
It looks like there are some snags/bugs (different ones!) in both implementations at this point. Even if existing bugs are fixed I still have situations where this pr list approach is preferred. Open to suggestions on how this PR can be improved in that sense.

How about adding the formatting of the user and the branch as done in the PR / issue details in order to provide some separation of the row information.

I think that sounds like a great idea regardless of all the above! That would solve the motivation of having author name viewable and filterable.  Though I guess that would mean that field/column-based sorting would become quite cumbersome?

Thanks for taking care of this PR @legobeat. It is still on draft state, are you planning on doing any more work on this PR?

Thanks for the update @legobeat We have created a discussion room in Matrix in case you want to join us https://matrix.to/#/#octo.nvim:matrix.org
Also, happy to make you a collaborator if you are up for it

authors: legobeat macovsky pwntester wd60622


text:
Weird behaviour when multiple tabs are open

Needs further investigation!

Haven’t managed to recreate this… I think it might just be a problem in development. Closing for the timebeing.

repo: pwyf/iati-org-viz


text:
[ENH] Add the ability to find Proper Possibly Directed Paths in a Graph

Is your feature request related to a problem? Please describe.
This is in continuation of PR #1148 in Dowhy. First PR in an effort to provide seamless integration with Dowhy.
Describe the solution you'd like
A couple of functions that can take a graphs and find Proper Possibly Directed Paths in a graph.

@adam2392 I have a question: The paper defines a proper directed path as follows: "A possibly directed path or possibly causal path from X to Y is a path from X to Y that does not contain an arrowhead pointing in the direction of X." Does this mean any edge (including undirected edge) is fine as long as an arrowhead doesn't point in the direction of X?

Yep!
These are all valid X ... -o ..., Y, or X ... o-o ..., Y, or ``X ... o-> ..., Y, or X ... -- ..., Y`. So the check should be quite easy.

if checking a single path: if the path from X to Y contains at any point a directed edge towards X, it is not a possibly directed path
if checking multiple paths: same as above

@adam2392 for this I will need to first know all the paths from X to Y regardless of edge type. Is there a function that already does that?

No, but one can possibly just convert to an undirected graph and then run: https://networkx.org/documentation/stable/reference/algorithms/generated/networkx.algorithms.simple_paths.all_simple_paths.html#all-simple-paths
Not "as efficient" cuz of the conversion, but we can just leave an in-line comment there.

@adam2392 continuing the conversation, if I just convert to an undirected graph and run "all_simple_paths" then I need to find a way to identify any edges that might contain an arrowhead pointing towards X. Once I convert to an undirected graph, I loose this information. One way is to do this is

convert all then graphs to undirected graphs and then run "all_simple_paths"
find all the arrowheads pointing towards X in all of the possible paths from X in the original graph. Then keep a map of all of these edges and check in the resulting path from step 1 if it contains any of these edges.

Yeah I think it's pretty inefficient, but the only other way I can think of is a BFS/DFS style algorithm that just keeps track of edge types along each path.
However, I'm okay with a more inefficient algorithm initially and an inline comment documenting why its inefficient.

Actually, I think the BFS/DFS style approach would be easier to implement. Since all I will have to do is check for a backwards arrow at each edge while keeping track of all the possible paths. For any path to be invalid, either it has a backwards arrow or it doesn't lead to Y. Otherwise, return all paths.

Of course, this will be more memory intensive since I will have to keep all the possible paths in memory, which makes me wonder whether this approach would in fact be worse for larger graphs.

On second thought, in method 1, both the steps are essentially BFS/DFS with all the paths being kept track of in memory. This would done the same memory intensive procedure twice.
And the step 2 in method 1 is essentially method 2.
I think we should go with the second method from the start.

Sounds good!

repo: py-why/pywhy-graphs


text:
specify number of datasets

Please humour me by accepting the proposed changes!

Testing out to reject.


text:
Question

Hello I just wanted to ask does the tetherfi need Internet/wifi to work or not?

Hello!
Yes, the tether shares your existing device Internet connection with other devices.
Your phone will run TetherFi, your other devices connect to your phone, and then any time your laptop wants to talk to the internet, the phone will "talk for it" basically, so the phone needs an Internet connection of some kind.
Hope that helps!

authors: Dgfdgtbchjhh pyamsoft

repo: pyamsoft/tetherfi


text:
Doc/fix graphviz

Fixing automatic Graphviz image generation in ci

example fixed:

repo: pyansys/pydpf-core


text:
Use delvewheel to repair wheels on Windows

Description
Adds what was missing to be done in #3783 – this should ensure that we get some extra UCRT DLLs in a wheel.libs folder inside the Windows wheel. This is not required but would be good to have because we use MSVC as our compiler
Type of change
Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

[x] Bug fix (non-breaking change which fixes an issue)

Key checklist:

[x] No style issues: $ pre-commit run (or $ nox -s pre-commit) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
[x] All tests pass: $ python run-tests.py --all (or $ nox -s tests)
[x] The documentation builds: $ python run-tests.py --doctest (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ python run-tests.py --quick (or $ nox -s quick).
Further checks:

[ ] Code is commented, particularly in hard-to-understand areas
[ ] Tests added that prove fix is effective or that feature works

Triggered a run at https://github.com/agriyakhetarpal/PyBaMM/actions/runs/9164324329/job/25195376665, it should pass most likely

repo: pybamm-team/PyBaMM


text:
Add summaries in frontmatter for all pages

The summary: section in the YAML frontmatter shows up in search engines as a brief description of the page. Most pages don't have this currently, which means that the same text gets displayed.
Every separate page should have its own summary, and other fields such as dates or etc.  that are permitted by Hugo parameters or YAML itself.

Thanks, @Bashamega. This will be helpful. For the summaries, we just need one sentence. Could you make your own?

Thanks, @Bashamega. This will be helpful. For the summaries, we just need one sentence. Could you make your own? We will be able to comment on those and fix them up if needed.

Ok, please assign me

authors: Bashamega agriyakhetarpal


text:
UIControl constants are undefined

The UIControl constants will be undefined unless we import them from libs/uikit.py

PR Checklist:

[x] All new features have been tested
[x] All new features have been documented
[x] I have read the CONTRIBUTING.md file
[x] I will abide by the code of conduct

Thanks!
Agreed. I'll go through and narrow it down.

repo: pybee/toga


text:
Default validator

continuing from #200, moving default and on_error behaviour into separate validator for reuse in other places.
@PrettyWood this is ready to review and will be ready to merge soon. But I know you're busy so I'll probably merge once I'm happy unless anyone brings up problems.
TODO:

[x] check performance doesn't get worse
[x] support positional tuples
[ ] coverage

Codecov Report

Merging #234 (0b430dd) into main (cea60f8) will decrease coverage by 0.40%.
The diff coverage is 93.42%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #234      +/-   ##
==========================================
- Coverage   98.65%   98.24%   -0.41%
==========================================
  Files          48       49       +1
  Lines        5057     5140      +83
  Branches       35       36       +1
==========================================
+ Hits         4989     5050      +61
- Misses         68       89      +21
- Partials        0        1       +1

Impacted Files
Coverage Δ

src/build_tools.rs
96.00% <0.00%> (-1.30%)
:arrow_down:

src/errors/line_error.rs
97.18% <0.00%> (-1.39%)
:arrow_down:

src/errors/validation_exception.rs
97.00% <0.00%> (-0.98%)
:arrow_down:

src/validators/date.rs
95.18% <0.00%> (-1.17%)
:arrow_down:

src/validators/with_default.rs
94.44% <94.44%> (ø)

src/validators/typed_dict.rs
98.61% <96.96%> (-0.29%)
:arrow_down:

pydantic_core/schema_types.py
99.16% <100.00%> (-0.84%)
:arrow_down:

src/validators/mod.rs
98.34% <100.00%> (+0.01%)
:arrow_up:

src/input/input_abstract.rs
96.47% <0.00%> (-3.53%)
:arrow_down:

src/errors/kinds.rs
96.63% <0.00%> (-2.53%)
:arrow_down:

... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cea60f8...0b430dd. Read the comment docs.

replaced by #242 to try to get codecov to work :angry:.

authors: codecov-commenter samuelcolvin

repo: pydantic/pydantic-core


text:
Cannot subclass from AnyUrl

Initial Checks

[X] I confirm that I'm using Pydantic V2 installed directly from the main branch, or equivalent

Description
With pydantic v1 I used to subclass the AnyUrl to slightly change its behavior in a custom class.
It seems that it is not possible anymore with pydantic v2. I literally cannot subclass it anymore.
Is this expected ?
Example Code
from pydantic import AnyUrl

class A(AnyUrl):
    pass

Python, Pydantic & OS Version
pydantic version: 2.0b2
        pydantic-core version: 0.38.0 release build profile
                 install path: /home/sami/snap/alacritty/common/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.9/lib/python3.9/site-packages/pydantic
               python version: 3.9.10 (main, Jan 18 2022, 17:20:31)  [GCC 9.3.0]
                     platform: Linux-5.15.0-73-generic-x86_64-with-glibc2.31
     optional deps. installed: ['typing-extensions']

This is expected, it's a Rust class, you won't be able to subclass it. What is it you are trying to do that requires you to subclass it?

I am currently adapting docarray to pydantic v2. There we define a class that inherits AnyUrl and slightly change the behavior: https://github.com/docarray/docarray/blob/main/docarray/typing/url/any_url.py
But I guess that we will have to rewrite this base class ourselves and don't rely on the built-in AnyUrl (or maybe reuse it internally without inheriting from it).

Yes and no: subclasses will have to override __new__ instead of __init__ for starters, which still makes a lot of use cases weird. Hence why I was asking @samsja what their actual goal was. I also think we should in general encourage composition over inheritance (especially when it can be done vía Annotated, which is almost always the case unless you need to add attributes, in which case you run into the __new__ thing I mentioned above). Finally I don’t know what impact that has on the generated Rust code (David H would know for sure) but since it’s not the default I imagine it’s not zero.
that said, if you still feel it’s worth it I don’t mind merging it.

On my side basically what I am looking for is:

being able to reuse some part of AnyUrl core logic. This can be done via composition over inheritance
inheriting the serilization ability of AnyUrl. Pydantic know how to deal with AnyUrl but will not know out of the box how to deal with my new class. (Though it can be solved pretty easily)

Inheriting from the class seems like the "easiest" way to do what I want to do, but it is not 100% needed honestly. Tho if there are no downside on pydantic side it could be nice to have it

FYI in V1 if you inherit from a class you inherit it’s validation and serialization. In V2 you do not.

FYI in V1 if you inherit from a class you inherit it’s validation and serialization. In V2 you do not.

hmmm interesting. Is there a way to not have to copy past the validation/ser ability of the class when inheriting ?

How about something like this?
from typing import Any

from pydantic_core import CoreSchema, core_schema

from pydantic import AnyUrl, BaseModel, GetCoreSchemaHandler, GetJsonSchemaHandler
from pydantic.json_schema import JsonSchemaValue

class MyUrl:
    def __init__(self, url: AnyUrl) -> None:
        self._url = url

    @classmethod
    def validate(
        cls,
        value: AnyUrl,
    ) -> 'MyUrl':
        assert value.host == 'example.com', 'Url must be example.com'
        return MyUrl(value)

    @classmethod
    def serialize(
        cls,
        value: 'MyUrl',
        handler: core_schema.SerializerFunctionWrapHandler,
    ) -> Any:
        return handler(value._url)

    @classmethod
    def __get_pydantic_core_schema__(cls, _source_type: Any, handler: GetCoreSchemaHandler) -> core_schema.CoreSchema:
        url_schema = handler.generate_schema(AnyUrl)
        return core_schema.no_info_after_validator_function(
            cls.validate, url_schema, serialization=core_schema.wrap_serializer_function_ser_schema(cls.serialize)
        )

    @classmethod
    def __get_pydantic_json_schema__(cls, core_schema: CoreSchema, _handler: GetJsonSchemaHandler) -> JsonSchemaValue:
        return {'type': 'string', 'format': 'uri'}

    def __repr__(self) -> str:
        return f'MyUrl({self._url})'

class Model(BaseModel):
    url: MyUrl

print(Model(url='https://example.com'))
print(Model(url='https://pydantic.dev'))
#> 1 validation error for Model:
#> Assertion failed, Url must be example.com

Or depending on what you're trying to do you could use something like this:
from typing import Annotated, Any

from pydantic import AnyUrl, BaseModel
from pydantic.functional_serializers import PlainSerializer
from pydantic.functional_validators import AfterValidator

def validate(
    value: AnyUrl,
) -> AnyUrl:
    assert value.host == 'example.com', 'Url must be example.com'
    return value

def serialize(
    value: AnyUrl,
) -> Any:
    return value.host

MyUrl = Annotated[AnyUrl, AfterValidator(validate), PlainSerializer(serialize, return_type=str)]

class Model(BaseModel):
    url: MyUrl

print(Model(url='https://example.com'))
print(Model(url='https://example.com').model_dump())
print(Model(url='https://pydantic.dev'))
#> 1 validation error for Model:
#> Assertion failed, Url must be example.com

Okay I see makes sense !
Btw where is the best way to get up to speed with pydantic v2 ? Reading your code showed me I am still missing some understanding of the new API of v2

We're actively working on documentation. Tests are probably the best documentation right now.

okay thanks !

authors: adriangb samsja

repo: pydantic/pydantic


text:
Add SerializeAsAny to migration guide and possibly move functionality into model_config

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
We ran into some issues with serialization when migrating (subclass data loss). It wasn't until finding this bug  that led us to discover the docs with the Migration warning  stating you need to use SerializeAsAny.
This is less a bug and more a request to put this front and center in the migration doc. We lost a lot of time trying to figure this out and given that serialization is so essential for any API, esp FastAPI, this isn't something that should require a ton of searching to figure out.
Anyway, wondering if there is a possible plan to implement this logic within the model_config = ConfigDict()? I think it would make more intuitive sense.
From Docs, perhaps add to migration guide:
This behavior is different from how things worked in Pydantic V1, where we would always include all (subclass) fields when recursively dumping models to dicts. The motivation behind this change in behavior is that it helps ensure that you know precisely which fields could be included when serializing, even if subclasses get passed when instantiating the object. In particular, this can help prevent surprises when adding sensitive information like secrets as fields of subclasses.
Thanks so much!
Example Code
No response
Python, Pydantic & OS Version
pydantic version: 2.1.0
        pydantic-core version: 2.4.0
          pydantic-core build: profile=release pgo=false mimalloc=true
                 install path: /Users/taddeskorris/.pyenv/versions/3.11.0/envs/merino-311/lib/python3.11/site-packages/pydantic
               python version: 3.11.0 (main, Dec 13 2022, 15:17:55) [Clang 13.1.6 (clang-1316.0.21.2.5)]
                     platform: macOS-12.5-arm64-arm-64bit
     optional deps. installed: ['typing-extensions']

Selected Assignee: @dmontagu

Anyway, wondering if there is a possible plan to implement this logic within the model_config = ConfigDict()? I think it would make more intuitive sense.

We decided it would be better as a runtime flag, see https://github.com/pydantic/pydantic-core/pull/740. The constraint has been bandwidth in implementation. A contribution to move that work forward or to improve the docs would be very welcome!

Closing as a duplicate of https://github.com/pydantic/pydantic/issues/6423, which we've included in our 2.6.0 milestone. Will remember to update the migration guide and docs as well 👍

authors: adriangb sydney-runkle taddes


text:
to_csv date_format including hh:mm:ss

Since 0.16.1 the to_csv output of a dataframe has started including hh:mm:ss with date_format strings like:
date_format='%d-%m-%Y'
Reverting back to 0.16.0 the date output was then formatted properly (without time).

You will have to be more specific and show an example. As this is as expected.
In [1]: pd.__version__
Out[1]: '0.16.1'

In [2]: DataFrame({'A' : pd.date_range('20130101',periods=5)}).to_csv()
Out[2]: ',A\n0,2013-01-01\n1,2013-01-02\n2,2013-01-03\n3,2013-01-04\n4,2013-01-05\n'

In [3]: DataFrame({'A' : pd.date_range('20130101',periods=5,freq='s')}).to_csv()
Out[3]: ',A\n0,2013-01-01 00:00:00\n1,2013-01-01 00:00:01\n2,2013-01-01 00:00:02\n3,2013-01-01 00:00:03\n4,2013-01-01 00:00:04\n'

There were some changes to correct a number of bugs in date formatting. This looks like an untested path. Marking as a bug.

this is releated to #7791

Actually, date_format appears to be altogether ignored in 0.16.1 to_csv:
In [18]: pd.__version__
Out[18]: '0.16.1'

In [19]: pd.DataFrame({'A' : pd.date_range('20130101',periods=5,freq='s')}).to_csv(date_format='%d-%m-%Y')
Out[19]: ',A\n0,2013-01-01 00:00:00\n1,2013-01-01 00:00:01\n2,2013-01-01 00:00:02\n3,2013-01-01 00:00:03\n4,2013-01-01 00:00:04\n'

In [4]: pd.__version__
Out[4]: '0.16.0'

In [5]: pd.DataFrame({'A' : pd.date_range('20130101',periods=5,freq='s')}).to_csv(date_format='%d-%m-%Y')
Out[5]: ',A\n0,01-01-2013\n1,01-01-2013\n2,01-01-2013\n3,01-01-2013\n4,01-01-2013\n'

closed by #10247

authors: blitzd jreback

repo: pydata/pandas


text:
PEP: pandas/core round 3 (generic, groupby, index)

yapf with manual modifications for error strings and bad yapf formatting.

No github diffs again!
There are some pieces of code that I think can be removed but want to confirm:
index.py i/j are not used anywhere.
groupby.py how can this work?  agg_func is not defined.

index i/j ok to remove

yeh that looks not tested, it for a transform on a Panel or greater. I think just raise NotImplemented (if the ndim >= 3) & remove the 2nd clause (where you noted). If nothing breaks in the test suite then ok!

I assume it should be transform_func(result[:, :, i], chunk, comp_ids, accum)

k, I can do that.

@rockg I would keep style changes and code clean-up (as in removal of unused code) strictly separated (at least in a separate commit). Just to be sure, in case we overlooked something, then it is not burried in a gigantic commit for which you can't see the diff.

The main goal of this exercise is to make flake8 passing and cleanup some style were necessary.  What I cleaned out were unused variables reported by flake8 and, as a result, I don't see a problem including that here.

@rockg I've been putting # noqa on these lines and adding a TODO comment to come back and investigate

This is especially important in unit tests where I suspect that there are some test cases that may have skipped some intended checks

Okay, that seems reasonable.

ok, rebasing core/groupby.py is going to be painful for #11841 but go ahead (my touches are mostly reorg anyhow)

Is this good to merge?

I'll leave this to @jreback with the pending rebase

if u want to remove groupby.py from this would be great (I'll catch that in resample fixes)
otherwise lgtm

thanks! we're almost there =)

authors: jorisvandenbossche jreback rockg wesm


text:
How to show sub headers on the left sidebar

I am trying to show sub headers in a page on the left sidebar. I can't find any option for that.

do you mean a subheader like the level 2 header from the pages?
If that's the case is it answering your problem?

number of navigation level shown by default: https://pydata-sphinx-theme.readthedocs.io/en/latest/user_guide/navigation.html#control-how-many-navigation-levels-are-shown-by-default
number of level shown in the collapsing btn: https://pydata-sphinx-theme.readthedocs.io/en/latest/user_guide/navigation.html#control-the-number-of-navigation-levels

do you mean a subheader like the level 2 header from the pages? If that's the case is it answering your problem?

number of navigation level shown by default: https://pydata-sphinx-theme.readthedocs.io/en/latest/user_guide/navigation.html#control-how-many-navigation-levels-are-shown-by-default
number of level shown in the collapsing btn: https://pydata-sphinx-theme.readthedocs.io/en/latest/user_guide/navigation.html#control-the-number-of-navigation-levels

I am try to show the items here on the left sidebar, basically as collapsed items under the topic.

like this ?
https://pydata-sphinx-theme.readthedocs.io/en/latest/user_guide/page-toc.html#show-more-levels-of-the-in-page-toc-by-default

like this ? https://pydata-sphinx-theme.readthedocs.io/en/latest/user_guide/page-toc.html#show-more-levels-of-the-in-page-toc-by-default

I tried that before, it didn't work. The left sidebar items were still the same.

just to make sure you are talking about the right sidebar ? (because your screenshot is showing the page TOC which is on the right side of the screen)
I tested the configuration in my test documentation and it worked as expected:
html_theme_options = {"show_toc_level": 3}

just to make sure you are talking about the right sidebar ? (because your screenshot is showing the page TOC which is on the right side of the screen)
I tested the configuration in my test documentation and it worked as expected:
html_theme_options = {"show_toc_level": 3}

I was using it as an example of what I want. I want the collapsible items to be on the left sidebar instead of the right. The way you have collapsible items for subpages.

you can't this theme only displays pages and subpages in the left sidebar and use the right sidebar to display in-page TOC

This theme is designed for heavy API that requires the header navbar to navigate their content. If you can afford to let the content of your pages in the left sidebar I assume it's way lighter, did you consider the other themes suggested in the documentation?

If you are looking for a Sphinx theme that puts all of its sub-pages in the sidebar, the Sphinx Book Theme has a similar look and feel, and Furo is another excellent choice. You can also see the Sphinx Themes Gallery for more ideas.

This theme is designed for heavy API that requires the header navbar to navigate their content. If you can afford to let the content of your pages in the left sidebar I assume it's way lighter, did you consider the other themes suggested in the documentation?

If you are looking for a Sphinx theme that puts all of its sub-pages in the sidebar, the Sphinx Book Theme has a similar look and feel, and Furo is another excellent choice. You can also see the Sphinx Themes Gallery for more ideas.

Thanks

repo: pydata/pydata-sphinx-theme


text:
DataArray.integrate has a 'dim' arg, but Dataset.integrate has a 'coord' arg

This is just a minor gripe but I think it should be fixed.
The API syntax is inconsistent:
ds.differentiate(coord='x')
da.differentiate(coord='x')
ds.integrate(coord='x')
da.integrate(dim='x')   # why dim??

It should definitely be coord - IMO it doesn't make sense to integrate or differentiate over a dim because a dim by definition has no information about the distance between grid points. I think because the distinction between dims and coords is one of the things that new users have to learn about, we should be strict to not confuse up the meanings in the documentation/API.
The discussion on the original PR seems to agree, so I think this was just an small oversight.
The only question is whether it requires a deprecation cycle?

Just found that @max-sixty already pointed this out.
It's bugging me, so I'll open a PR :)

repo: pydata/xarray


text:
Support for duck Dask Arrays

https://github.com/pydata/xarray/issues/525#issuecomment-531603357 raised the idea of adding "duck Dask Array" support to xarray as a way to handle xarray > Pint Quantity > Dask Array wrapping in a way that still allowed for most of xarray's Dask integration to work properly. With @rpmanser working on implementing the Dask collection interface in Pint (https://github.com/hgrecco/pint/pull/1129), I thought it best to elevate this to its own issue to track progress and discuss implementation on xarray's side (since hopefully @rpmanser or I can get started on it soon).
Two initial (and intertwined) discussion points that I'd like to bring up (xref https://github.com/hgrecco/pint/pull/1129#issuecomment-655197079):

How should xarray check for a duck Dask Array?
Is it acceptable for a Pint Quantity to always have the Dask collection interface defined (i.e., be a duck Dask array), even when its magnitude (what it wraps) is not a Dask Array?

cc @keewis, @shoyer, @crusaderky

Maybe something like this would work?
def is_duck_dask_array(x):
  return getattr(x, 'chunks', None) is not None

xarray.DataArray would pass this test (chunks is either None for non-dask arrays or a tuple for dask arrays), so this would be consistent with what we already do.

Maybe something like this would work?
def is_duck_dask_array(x):
  return getattr(x, 'chunks', None) is not None

xarray.DataArray would pass this test (chunks is either None for non-dask arrays or a tuple for dask arrays), so this would be consistent with what we already do.

That would be a straightforward solution to both problems! Now that https://github.com/hgrecco/pint/pull/1129 is merged, a Pint Quantity containing a Dask Array passes along the chunks attribute from the Dask Array, and a Pint Quantity containing something else will raise an AttributeError. Unless there are other objections, I'll see what it will take to swap out the existing Dask checks for this in the xarray internals and hopefully get around to a PR.

It might also make sense to check for one or more of the special dask collection attributes (__dask_graph__, __dask_keys__, etc)

Is it acceptable for a Pint Quantity to always have the Dask collection interface defined (i.e., be a duck Dask array), even when its magnitude (what it wraps) is not a Dask Array?

I think there are already enough headaches with __iter__ being always defined and confusing libraries such as pandas (https://github.com/hgrecco/pint/issues/1128).
I don't see why pint should be explicitly aware of dask (except in unit tests)? It should only deal with generic NEP18-compatible libraries (numpy, dask, sparse, cupy, etc.).

How should xarray check for a duck Dask Array?

We should ask the dask team to formalize what defines a "dask-array-like", like they already did with dask collections, and implement their definition in xarray.

I think there are already enough headaches with __iter__ being always defined and confusing libraries such as pandas (hgrecco/pint#1128).
I don't see why pint should be explicitly aware of dask (except in unit tests)? It should only deal with generic NEP18-compatible libraries (numpy, dask, sparse, cupy, etc.).

Since Pint wraps Dask, in order to leverage Dask Array functionality on Pint Quantities, we need to have the Dask collection interface available. In a sense, Pint needs special handling for Dask like xarray Variables do since they both can be upcast types of Dask Array. Implicitly passing through attributes (how Pint handles special methods/attributes of downcast types in general) from the wrapped Dask Array is not sufficient, however, because the finalizers have to rewrap with Quantity (see https://github.com/hgrecco/pint/pull/1129/files#diff-d9924213798d0fc092b8cff13928d747R1947-R1950), hence the explicit awareness of Dask being needed in Pint.

We should ask the dask team to formalize what defines a "dask-array-like", like they already did with dask collections, and implement their definition in xarray.

Done! See https://github.com/dask/dask/issues/6385.

Based on @mrocklin's comment in https://github.com/dask/dask/issues/6385, the plan will be to check for duck Dask Arrays with dask.is_collection along with xarray's previously used duck array check.

I can go ahead with putting together a PR for this. Before I do so, I'd like to clarify what is expected.

Implement the is_duck_dask_array() function
In that implementation, use dask.base.is_dask_collection() and the existing duck array check(s)
Replace isinstance(x, dask.array.Array) checks with the new is_dask_duck_array() function

I searched for existing duck array checks in xarray and nothing immediately obvious to me is showing up. It looks like a check for __array_function__ is inappropriate based on discussion in #3917. Could someone point out the proper duck array check?

We have https://github.com/pydata/xarray/blob/master/xarray/core/pycompat.py which defines dask_array_type and sparse_array_type and then use isinstance(da, dask_array_type) in a bunch of places (e.g. duck_array_ops).
re duck array check: @keewis added this recently
https://github.com/pydata/xarray/blob/f3ca63a4ac5c091a92085b477a0d34c08df88aa6/xarray/core/utils.py#L250-L253

@rpmanser That sounds like a good plan to me at least, but it would be great if any of the xarray maintainers would be able to chime in. Also, thanks again for being willing to work on this while I try working on https://github.com/dask/dask/issues/4583. The hidden 4th step is of course testing--primarily that this doesn't break existing functionality, but also that it works for duck Dask Arrays other than Dask Arrays themselves (not sure if Pint Quantities in upcoming v0.15 or a mocked class would be better).

Does/should any of this also consider #4212 ?

Does/should any of this also consider #4212 (CuPy)?

Only indirectly, since this deals with duck Dask arrays (things like Pint that go between xarray and Dask) rather than Dask chunks, which CuPy would be. But, once this, #4212, https://github.com/hgrecco/pint/issues/964, and https://github.com/dask/dask/pull/6393 are all in place, then we can test if xarray( pint( dask( cupy ))) works automatically from it all or not.

A couple of things came up in #4221

how do we ask a duck dask array to rechunk itself? pint seems to forward the .rechunk call but that isn't formalized anywhere AFAICT.
less important: should duck dask arrays cache their token somewhere? dask.array uses .name to do this and xarray uses that to check equality cheaply. We can use tokenize of course. But I'm wondering if it's worth asking duck dask arrays to cache their token as an optimization.

In Xarray we implemented the Dask collection spec.  https://docs.dask.org/en/latest/custom-collections.html#the-dask-collection-interface
We might want to do that with Pint as well, if they're going to contain Dask things.  That way Dask operations like dask.persist, dask.visualize, and dask.compute will work normally.

My guess is that we could steal the xarray.DataArray implementations over to Pint without causing harm.

In Xarray we implemented the Dask collection spec. https://docs.dask.org/en/latest/custom-collections.html#the-dask-collection-interface
We might want to do that with Pint as well, if they're going to contain Dask things. That way Dask operations like dask.persist, dask.visualize, and dask.compute will work normally.

That's exactly what's been done on Pint (see https://github.com/hgrecco/pint/pull/1129)!

That's exactly what's been done in Pint (see hgrecco/pint#1129)! @dcherian's points go beyond just that and address what Pint hasn't covered yet through the standard collection interface.

Ah, great.  My bad.

how do we ask a duck dask array to rechunk itself? pint seems to forward the .rechunk call but that isn't formalized anywhere AFAICT.

I think that you would want to make a pint array rechunk method that called down to the dask array rechunk method.  My guess is that this might come up in other situations as well.

less important: should duck dask arrays cache their token somewhere? dask.array uses .name to do this and xarray uses that to check equality cheaply. We can use tokenize of course. But I'm wondering if it's worth asking duck dask arrays to cache their token as an optimization.

I think that implementing the dask.base.normalize_token method should be fine.  This will probably be very fast because you're probably just returning the name of the underlying dask array as well as the unit of the pint array/quatity.  I don't think that caching would be necessary here.
It's also possible that we could look at the __dask_layers__ method to get this information.  My memory is a bit fuzzy here though.

Re:rechunk, this should be part of the spec I guess. We need this for DataArray.chunk().
xarray does do some automatic rechunking in variable.py. But this comment:
            # chunked data should come out with the same chunks; this makes
            # it feasible to combine shifted and unshifted data
            # TODO: remove this once dask.array automatically aligns chunks

suggest that we could delete that automatic rechunking today.

This will probably be very fast because you're probably just returning the name of the underlying dask array as well as the unit of the pint array/quatity.

ah yes, we can rely on the underlying array library to optimize this.

Dask collections tokenize quickly.  We just use the name I think.

authors: crusaderky dcherian dopplershift jthielen mrocklin rpmanser shoyer


text:
Cannot round-trip DataArray with string coordinates using numcodecs.Categorize filter

What happened?
Writing an array containing string coordinate values using the Numcodecs.Categorize filter succeeds, but Zarr group cannot be read back into a DataArray.
What did you expect to happen?
To get back the same array that I wrote.
Minimal Complete Verifiable Example
import numcodecs
import xarray as xr

da = xr.DataArray(coords={"x": ("x", np.array(["a", "b"], dtype=object))}, dims=("x",))
codec = numcodecs.Categorize(labels=["a", "b"], dtype=object)
encoding = {
    "x": {
        "filters": [codec],
    },
}

da.to_zarr("/tmp/foo.zarr", mode="w", encoding=encoding)
da = xr.open_dataarray("/tmp/foo.zarr")  # crashes

MVCE confirmation

[X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
[X] Complete example — the example is self-contained, including all data and the text of any traceback.
[X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
[X] New issue — a search of GitHub Issues suggests this is not a duplicate.
[X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output
Traceback (most recent call last):
  File "/home/yang.yang/foo.py", line 58, in <module>
    da = xr.open_dataarray("/tmp/foo.zarr")
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/backends/api.py", line 851, in open_dataarray
    dataset = open_dataset(
        filename_or_obj,
    ...<15 lines>...
        **kwargs,
    )
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/backends/api.py", line 670, in open_dataset
    backend_ds = backend.open_dataset(
        filename_or_obj,
    ...<2 lines>...
        **kwargs,
    )
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/backends/zarr.py", line 1524, in open_dataset
    ds = store_entrypoint.open_dataset(
        store,
    ...<6 lines>...
        decode_timedelta=decode_timedelta,
    )
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/backends/store.py", line 59, in open_dataset
    ds = Dataset(vars, attrs=attrs)
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/core/dataset.py", line 746, in __init__
    variables, coord_names, dims, indexes, _ = merge_data_and_coords(
                                               ~~~~~~~~~~~~~~~~~~~~~^
        data_vars, coords
        ^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/core/dataset.py", line 459, in merge_data_and_coords
    return merge_core(
        [data_vars, coords],
    ...<5 lines>...
        skip_align_args=[1],
    )
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/core/merge.py", line 699, in merge_core
    collected = collect_variables_and_indexes(aligned, indexes=indexes)
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/core/merge.py", line 362, in collect_variables_and_indexes
    idx, idx_vars = create_default_index_implicit(variable)
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/core/indexes.py", line 1425, in create_default_index_implicit
    index = PandasIndex.from_variables(dim_var, options={})
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/core/indexes.py", line 654, in from_variables
    obj = cls(data, dim, coord_dtype=var.dtype)
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/core/indexes.py", line 589, in __init__
    index = safe_cast_to_index(array)
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/core/indexes.py", line 469, in safe_cast_to_index
    index = pd.Index(np.asarray(array), **kwargs)
                     ~~~~~~~~~~^^^^^^^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/core/indexing.py", line 514, in __array__
    return np.asarray(self.get_duck_array(), dtype=dtype, copy=copy)
                      ~~~~~~~~~~~~~~~~~~~^^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/backends/common.py", line 268, in get_duck_array
    return self[key]  # type: ignore[index]
           ~~~~^^^^^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/backends/zarr.py", line 226, in __getitem__
    return indexing.explicit_indexing_adapter(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        key, array.shape, indexing.IndexingSupport.VECTORIZED, method
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/core/indexing.py", line 1018, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/xarray/backends/zarr.py", line 216, in _getitem
    return self._array[key]
           ~~~~~~~~~~~^^^^^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/zarr/core.py", line 797, in __getitem__
    result = self.get_basic_selection(pure_selection, fields=fields)
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/zarr/core.py", line 923, in get_basic_selection
    return self._get_basic_selection_nd(selection=selection, out=out, fields=fields)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/zarr/core.py", line 965, in _get_basic_selection_nd
    return self._get_selection(indexer=indexer, out=out, fields=fields)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/zarr/core.py", line 1340, in _get_selection
    self._chunk_getitems(
    ~~~~~~~~~~~~~~~~~~~~^
        lchunk_coords,
        ^^^^^^^^^^^^^^
    ...<4 lines>...
        fields=fields,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/zarr/core.py", line 2185, in _chunk_getitems
    self._process_chunk(
    ~~~~~~~~~~~~~~~~~~~^
        out,
        ^^^^
    ...<6 lines>...
        partial_read_decode=partial_read_decode,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/zarr/core.py", line 2098, in _process_chunk
    chunk = self._decode_chunk(cdata)
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/zarr/core.py", line 2361, in _decode_chunk
    chunk = f.decode(chunk)
  File "numcodecs/vlen.pyx", line 141, in numcodecs.vlen.VLenUTF8.decode
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/numcodecs/compat.py", line 149, in ensure_contiguous_ndarray
    ensure_contiguous_ndarray_like(buf, max_buffer_size=max_buffer_size, flatten=flatten)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yang.yang/.venv/lib/python3.13/site-packages/numcodecs/compat.py", line 99, in ensure_contiguous_ndarray_like
    raise TypeError("object arrays are not supported")
TypeError: object arrays are not supported

Anything else we need to know?
No response
Environment

INSTALLED VERSIONS
commit: None
python: 3.13.0 (main, Oct 16 2024, 03:23:02) [Clang 18.1.8 ]
python-bits: 64
OS: Linux
OS-release: 6.8.0-1017-aws
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('C', 'UTF-8')
libhdf5: None
libnetcdf: None
xarray: 2024.11.0
pandas: 2.2.3
numpy: 2.1.3
scipy: None
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
zarr: 2.18.3
cftime: None
nc_time_axis: None
iris: None
bottleneck: None
dask: 2024.12.0
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2024.10.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: None
pip: None
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None

After doing a little digging, I believe the issue is that when the string object dtype is "object" (as opposed to say <U3), the "VLenUTF8" filter is automatically added during the write. The result of the variable-length encoding is a single byte array, which when passed as the input to the Categorize codec, causes all of the values to "miss". On the read back, the Categorize filter transforms the data to object (all empty strings), which is not what the VLenUTF8 filter is expecting.
import numcodecs
import numpy as np
import xarray as xr

da = xr.DataArray(coords={"x": ("x", np.array(["a", "b", "c"], dtype="object"))}, dims=("x",))

c1 = numcodecs.VLenUTF8()
x = c1.encode(da.x.data)
print(repr(x))    # bytearray(b'\x03\x00\x00\x00\x01\x00\x00\x00a\x01\x00\x00\x00b\x01\x00\x00\x00c')

c2 = numcodecs.Categorize(labels=da.x.data, dtype=object)
y = c2.encode(x)  # No matches, since x is a byte array
print(repr(y))  # array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=uint8)

z = c2.decode(y)
print(repr(z))  # array(['', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', ''], dtype=object)

c1.decode(z)  # boom

Is there anything we can do about this? Being able to encode variable-length data has be pretty useful, but doesn't seem to play well with the cascading level of filters.


text:
Error on using dynamic loading

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/home/vscode/.local/lib/python3.11/site-packages/watchdog/observers/api.py", line 204, in run
    self.dispatch_events(self.event_queue)
  File "/home/vscode/.local/lib/python3.11/site-packages/watchdog/observers/api.py", line 380, in dispatch_events
    handler.dispatch(event)
  File "/home/vscode/.local/lib/python3.11/site-packages/watchdog/events.py", line 276, in dispatch
    {
  File "/home/vscode/.local/lib/python3.11/site-packages/edifice/runner.py", line 119, in on_modified
    if isinstance(observer, FSEventsObserver) and event.is_directory:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union

Looks like importing watchdog.observers.fsevents.FSEventsObserver caused ModuleNotFoundError somehow (I don't know why so I added it to issue of watchdog: https://github.com/gorakhargosh/watchdog/issues/1008) and messed up
try:
    from watchdog.observers.fsevents import FSEventsObserver
except ImportError:
    FSEventsObserver = None

in runner.py.
I monkey-patched the code like this -
        def on_modified(self, event):
            if FSEventsObserver is not None and isinstance(observer, FSEventsObserver) and event.is_directory:

(line 117-118) and it worked properly
Maybe something related to my installation (I install this on codespace with devcontainer and quite a lot of apt install mess) or installation of watchdog went something wrong.
(Oh and plus - this project doesn't requires watchdog in requirements.txt which made me install watchdog manually, would be nice if it is installed along with edifice when I just pip install pyedifice.)

dupe of https://github.com/pyedifice/pyedifice/issues/18. Sorry.

repo: pyedifice/pyedifice


text:
Better separation of Documentation and Concepts

General

pyfar version: 0.2.x

Description
There are parts of the documentation that pertain to concepts of pyfar rather than documenting functions or classes. Examples are:

Concept of Energy and Power signals
Explanation of coordinates systems
Explanations of FFT related stuff
Explanations of signal arithmetics

A possibility to speparate might be to add a section 'Concepts' to the docs. Parts of these section can be linked from the real documentation without loosing information and making the concepts easier to access at the same time.

In my opinion this is a great idea. I think your suggestion of adding a concepts section to the docs would be a good help for people just getting started with using pyfar.

authors: al-de f-brinkmann

repo: pyfar/pyfar


text:
Parsing time from logfile for training/testing Loss and Ladder steps

Added two functions to parse the log.txt file from pacemaker to get the time

for loss calculations for both training and testing (when present) datasets
for each ladder step when ladder scheme is being used

Pull Request Test Coverage Report for Build 11708844815
Details

0 of 0   changed or added relevant lines in 0 files are covered.
148 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.02%) to 3.681%

Files with Coverage Reduction
New Missed Lines
%

pacemaker/job.py
148
0.0%

Totals

Change from base Build 11700044535:
-0.02%

Covered Lines:
193

Relevant Lines:
5243

💛  - Coveralls

authors: HaithamGaafer coveralls


text:
Add tests?

Hi, this question is quite selfish, I'd like to use this repo among others for smoke testing new pydantic releases, see https://github.com/pydantic/pydantic/discussions/4359. According to this this repo has lots of pydantic imports, hence why I came here.
Would you consider adding tests to this repo, I don't know if there's a dummy api for telegram, otherwise you could create mock endpoints for unit tests.
Do you have an existing strategy for testing?

Hi, I don't have any strategy for testing yet.
Telegram needs you to be authorized to use API, so it's quite a problem to run tests automatically, in GitHub actions for example. But mock endpoints could help in this situation.
I will think about adding tests. Any PRs are welcome

Thanks for considering it.
In the absence of tests, I'd love it if you could test v1.10.0a1, and let us know if you see any breaking changes?

authors: pylakey samuelcolvin

repo: pylakey/aiotdlib


text:
Parameter recovery on simulated data - issue with saturation parameter

Hello,
I'm building an model and trying to recover some parameters that I set through data simulation. I'm currently having issues recovering the true parameters aside from the adstock. I'm working with geometric adstock and hill saturation. I've done some testing with varying number of variables (2-4 attached). I've done 3 runs on each notebook attached. An iteration with uninformative priors, informative priors on shape of the curve, informative priors on adstock and shape of the curve. Through all of my testing, including iterations outside of the attached notebooks, adstock is the only parameter I've consistently recovered.
I would appreciate some direction or insight on where I can improve.
link to notebook on github which contains the same code and output as below:
https://github.com/jaylin1/MMM/blob/main/Parameter_recovery - 4 vars-Copy1 (1).ipynb

 code
import pandas as pd
import numpy as np
import pymc as pm
import arviz as az
import matplotlib.pyplot as plt
import pymc.sampling_jax
from sklearn.preprocessing import MaxAbsScaler, StandardScaler
from pymc_marketing.mmm.transformers import geometric_adstock
from collections import defaultdict

data generating
seed: int = sum(map(ord, "mmm"))
rng: np.random.Generator = np.random.default_rng(seed=seed)

# date range
min_date = pd.to_datetime("2018-04-01")
max_date = pd.to_datetime("2021-09-01")

df = pd.DataFrame(
    data={"date_week": pd.date_range(start=min_date, end=max_date, freq="W-MON")}
).assign(
    year=lambda x: x["date_week"].dt.year,
    month=lambda x: x["date_week"].dt.month,
    dayofyear=lambda x: x["date_week"].dt.dayofyear,
)
n = df.shape[0]

x1 = rng.uniform(low=0.0, high=1.0, size=n)
df["x1"] = np.where(x1 > 0.9, x1, x1 / 2)
x2 = rng.uniform(low=0.0, high=1.0, size=n)
df["x2"] = np.where(x2 > 0.8, x2, 0)
x3 = rng.uniform(low=0.0, high=1.0, size=n)
df["x3"] = np.where(x3 > 0.6, x3, x3/3)
x4 = rng.uniform(low=0.0, high=1.0, size=n)
df["x4"] = np.where(x4 > 0.7, x4, x4 / 1.5)

alpha1: float = 0.7
alpha2: float = 0.3
alpha3: float = 0.5
alpha4: float = 0.4

df["x1_adstock"] = (
    geometric_adstock(x=df["x1"].to_numpy(), alpha=alpha1, l_max=8, normalize=True)
    .eval()
    .flatten()
)

df["x2_adstock"] = (
    geometric_adstock(x=df["x2"].to_numpy(), alpha=alpha2, l_max=8, normalize=True)
    .eval()
    .flatten()
)

df["x3_adstock"] = (
    geometric_adstock(x=df["x3"].to_numpy(), alpha=alpha3, l_max=8, normalize=True)
    .eval()
    .flatten()
)

df["x4_adstock"] = (
    geometric_adstock(x=df["x4"].to_numpy(), alpha=alpha4, l_max=8, normalize=True)
    .eval()
    .flatten()
)

def hill_saturation(x, lam, k):
    return  (x**k / (x**k + lam**k))

lam1: float = 0.45
lam2: float = 1.0
lam3: float = 0.5
lam4: float = 0.6

k1 : float = .5
k2 : float  = .7
k3 : float  = .9
k4 : float  = 1.5

df["x1_adstock_saturated"] = hill_saturation(df["x1_adstock"].to_numpy(), lam1, k1)
df["x2_adstock_saturated"] = hill_saturation(df["x2_adstock"].to_numpy(), lam2, k2)
df["x3_adstock_saturated"] = hill_saturation(df["x3_adstock"].to_numpy(), lam3, k3)
df["x4_adstock_saturated"] = hill_saturation(df["x4_adstock"].to_numpy(), lam4, k4)

df["intercept"] = 2.0
df["epsilon"] = rng.normal(loc=0.0, scale=0.25, size=n)

amplitude = 1
beta_1 = 4.0
beta_2 = 2.0
beta_3 = 2.5
beta_4 = 3.5

df["y"] = amplitude * (
    df["intercept"]
    + beta_1 * df["x1_adstock_saturated"]
    + beta_2 * df["x2_adstock_saturated"]
    + beta_3 * df["x3_adstock_saturated"]
    + beta_4 * df["x4_adstock_saturated"]
    + df["epsilon"]
)

scaling and prior setting
channels_exp_lst = ['x1','x2', 'x3' , 'x4']
target_name = 'y'

total_spend_per_channel = df[['x1', 'x2', 'x3' , 'x4']].sum(axis=0)
spend_share = total_spend_per_channel / total_spend_per_channel.sum()
n_channels = len(channels_exp_lst)
prior_sigma = n_channels * spend_share.to_numpy()
prior_sigma.tolist()

def scale_data(df, predictor_lst, target_name):

    sclr_trgt = MaxAbsScaler()
    sclr_predictor = MaxAbsScaler()
    sclr_control = StandardScaler()
    data_to_model = df.copy()

    sclr = defaultdict()
    # Scale Target
    sclr['transformer_target'] = MaxAbsScaler().fit(data_to_model[target_name].to_numpy().reshape(-1,1))
    data_to_model[target_name] = sclr['transformer_target'].transform(data_to_model[target_name].to_numpy().reshape(-1,1))

    # Scale Predictors with MaxAbs
    sclr['transformer_predictor'] = MaxAbsScaler().fit(data_to_model[predictor_lst].to_numpy())
    data_to_model[predictor_lst] = sclr['transformer_predictor'].transform(data_to_model[predictor_lst].to_numpy())

    return data_to_model, sclr

data_sclr, sclr = scale_data(df,  channels_exp_lst, target_name)

data_to_model_channel = data_sclr.melt(
  id_vars= ['date_week', target_name],
    value_vars = channels_exp_lst,
    ).rename(columns= {'variable': 'channel', 'value':'EXP'},
            ).pivot_table(
                    index= ['date_week', 'channel'],
                    values= ['EXP', target_name]
                    )

data_to_model_channel = data_to_model_channel.reset_index()
data_to_model_channel['channel'] = pd.Categorical(
    data_to_model_channel['channel'],
    categories=channels_exp_lst,
    ordered=True
)

data_to_model_channel = data_to_model_channel.sort_values(by=['date_week', 'channel'])
data_to_model_channel = data_to_model_channel.set_index(['date_week', 'channel'])

Modeling
COORDS = {
    'date_week' : df['date_week'].unique(),
    'channel' : channels_exp_lst,
}

with pm.Model(coords = COORDS) as model:

    geometric_adstock = GeometricAdstock(l_max= 8, normalize=True)

    x_data = pm.Data(
        "x_data",
        value= data_sclr[channels_exp_lst].values,
        dims=("date_week", "channel")
    )
    y_data = pm.Data(
        "y_data",
        value= data_sclr['y'].values,
        dims= ("date_week" )
    )

    int_sigma = pm.HalfNormal("int_sigma", sigma= .5)
    intercept = pm.HalfNormal("intercept", sigma= int_sigma,)
    beta = pm.HalfNormal('beta',  sigma= prior_sigma, dims = ['channel'])

    x1_alpha = pm.Uniform('x1_alpha', lower = .6, upper = .8)
    x2_alpha = pm.Uniform('x2_alpha', lower = .2, upper = .4)
    x3_alpha = pm.Uniform('x3_alpha', lower = .3, upper = .8)
    x4_alpha = pm.Uniform('x4_alpha', lower = .3, upper = .8)
    adstock_alpha = pm.Deterministic('adstock_alpha', pm.math.stack([x1_alpha, x2_alpha, x3_alpha, x4_alpha]), dims = ["channel"])

    sat_lam = pm.Uniform('sat_lam', lower = 0.1, upper = 1, dims = ['channel'])

    x1_sat_k = pm.Uniform('x1_sat_k', lower = 0.3, upper = 3.0)
    x2_sat_k = pm.Uniform('x2_sat_k', lower = 0.3, upper = 3.0)
    x3_sat_k = pm.Uniform('x3_sat_k', lower = 0.3, upper = 3.0)
    x4_sat_k = pm.Uniform('x4_sat_k', lower = 0.3, upper = 3.0)
    sat_k = pm.Deterministic('sat_k', pm.math.stack([x1_sat_k, x2_sat_k , x3_sat_k, x4_sat_k]), dims = ["channel"])

    adstock_data = geometric_adstock.function(x= x_data, alpha= adstock_alpha)
    saturated_data = hill_saturation(x= adstock_data,
                                         lam = sat_lam,
                                         k = sat_k
                                        )
    channel_contributions = pm.Deterministic(
        name= "channel_contributions",
        var = (saturated_data * beta) ,
        dims= ("date_week", "channel")
    )
    y_pred = intercept  + (channel_contributions).sum(axis=1)
    err_sigma= pm.HalfNormal('err_sigma', sigma= 1)

    pm.Normal(
        name = target_name, mu = y_pred,
        sigma = err_sigma,
        dims= ("date_week"),
        observed= y_data)

    ppc_mmm = pm.sample_prior_predictive(samples = 100, random_seed=3333)
    fig, ax = plt.subplots()
    az.plot_ppc(ppc_mmm, group="prior", num_pp_samples=100, ax = ax)
    ax.set_xlim([-5, 5])
    ax.set_ylim([0, 5])
    plt.show()

    idata_mmm = pm.sample(1000, tune=1000)
    pm.sample_posterior_predictive(idata_mmm, extend_inferencedata= True, random_seed= 123)

Visualizing actual curve vs estimated curve
curve_params = az.summary(idata_mmm, var_names=['sat_lam', 'sat_k'])
lam1_pred = curve_params.loc['sat_lam[x1]', 'mean']
lam2_pred = curve_params.loc['sat_lam[x2]', 'mean']
lam3_pred = curve_params.loc['sat_lam[x3]', 'mean']
lam4_pred = curve_params.loc['sat_lam[x4]', 'mean']
# lam5_pred = curve_params.loc['sat_lam[x5]', 'mean']
k1_pred = curve_params.loc['sat_k[x1]', 'mean']
k2_pred = curve_params.loc['sat_k[x2]', 'mean']
k3_pred = curve_params.loc['sat_k[x3]', 'mean']
k4_pred = curve_params.loc['sat_k[x4]', 'mean']
# k5_pred = curve_params.loc['sat_k[x5]', 'mean']

# actual parameters
# lam1: float = 0.45
# lam2: float = 1.0
# lam3: float = 0.5
# lam4: float = 0.6

# k1 : float = .5
# k2 : float  = .7
# k3 : float  = .9
# k4 : float  = 1.5

x = np.linspace(0.1, 2, 100)
y1 = hill_saturation(x, lam1, k1)
y2 = hill_saturation(x, lam2, k2)
y3 = hill_saturation(x, lam3, k3)
y4 = hill_saturation(x, lam4, k4)

pred_y1 = hill_saturation(x, lam1_pred, k1_pred)
pred_y2 = hill_saturation(x, lam2_pred, k2_pred)
pred_y3 = hill_saturation(x, lam3_pred, k3_pred)
pred_y4 = hill_saturation(x, lam4_pred, k4_pred)

plt.figure(figsize=(10, 10))
plt.plot(x, y1, label="x1 Actual (lam=0.45, k=1.0)", color='blue', linestyle='dotted')
plt.plot(x, y2, label="x2 Actual (lam=1.0, k=1.5)", color='red', linestyle='dotted')
plt.plot(x, y3, label="x3 Actual (lam=0.5, k=2.0)", color='orange', linestyle='dotted')
plt.plot(x, y4, label="x4 Actual (lam=0.6, k=2.5)", color='green', linestyle='dotted')

plt.plot(x, pred_y1, label=f"x1 Predicted (lam={lam1_pred}, k={k1_pred})", color='blue')
plt.plot(x, pred_y2, label=f"x2 Predicted (lam={lam2_pred}, k={k2_pred})", color='red')
plt.plot(x, pred_y3, label=f"x3 Predicted (lam={lam3_pred}, k={k3_pred})", color='orange')
plt.plot(x, pred_y4, label=f"x4 Predicted (lam={lam4_pred}, k={k4_pred})", color='green')

plt.xlabel('x')
plt.ylabel('y')
plt.title('Saturation Curves: Actual vs Predicted')
plt.legend()
plt.show()

def calculate_mape(actual, predicted):
    return np.abs((actual - predicted) / actual) * 100

lam_preds = [lam1_pred, lam2_pred, lam3_pred, lam4_pred]
lam_actuals = [lam1, lam2, lam3, lam4]
lam_mape = [calculate_mape(lam_actuals[i], lam_preds[i]) for i in range(4)]
lam_mape_df = pd.DataFrame({
    'lam': [f'lam{i+1}' for i in range(4)],
    'actual': lam_actuals,
    'pred' : lam_preds,
    'mape': lam_mape
})

k_preds = [k1_pred, k2_pred, k3_pred, k4_pred]
k_actuals = [k1, k2 , k3, k4]
k_mape = [calculate_mape(k_actuals[i], k_preds[i]) for i in range(4)]
k_mape_df = pd.DataFrame({
    'k': [f'k{i+1}' for i in range(4)],
    'actual': k_actuals,
    'pred' : k_preds,
    'mape': k_mape
})

I suggest you first do this with one channel and NOT SCALING anything first, as you have to be careful with parameter transformation. Please take a detailed look at https://juanitorduz.github.io/pymc_mmm/ where I explain this in detail

Hi @jaylin1,
I am looking over your code snippet.

Hi @jaylin1
Any progress with this?

repo: pymc-labs/pymc-marketing


text:
Downgrade the python version for readthedocs

Should fix failing action from #105

I used the old way of setting the python version, sorry. It should be like https://github.com/arviz-devs/xarray-einstats/blob/1e032006d32c145a832d1fb25d2e5d9e48ecdafb/.readthedocs.yaml#L5

authors: OriolAbril lucianopaz


text:
pyocd 0.29.0: TypeError with '&' operation in dap_access_cmsis_dap.py

We've seen a case when command line pyocd-flashtool -d debug -t k64f -f 3000000 -O connect_mode=under-reset --board 0240000034544e45004d00028aa900222011000097969900 /var/lib/lava/dispatcher/tmp/2379758/deployimages-dl229rxw/zephyr/zephyr.bin leads to:
Traceback (most recent call last):
  File "/usr/local/bin/pyocd-flashtool", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/pyocd/tools/flash_tool.py", line 161, in main
    with session:
  File "/usr/local/lib/python3.7/dist-packages/pyocd/core/session.py", line 343, in __enter__
    self.open()
  File "/usr/local/lib/python3.7/dist-packages/pyocd/core/session.py", line 457, in open
    self._probe.open()
  File "/usr/local/lib/python3.7/dist-packages/pyocd/probe/cmsis_dap_probe.py", line 148, in open
    self._link.open()
  File "/usr/local/lib/python3.7/dist-packages/pyocd/utility/concurrency.py", line 28, in _locking
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pyocd/probe/pydapaccess/dap_access_cmsis_dap.py", line 608, in open
    self._has_swo_uart = (self._capabilities & Capabilities.SWO_UART) != 0
TypeError: unsupported operand type(s) for &: 'str' and 'int'

That's definitely not a common situation (first time I see that), and it likely happens in extreme conditions (we run it on RaspberryPi4 which is not as stable with USB connections as a typical desktop computer, and run against board which has kind of worn-out flash). But the bottom line is the same: self._capabilities above may unexpectedly end up with a string instead of int.
Detailed run log for reference: https://validation.linaro.org/scheduler/job/2379758#L35

Thanks for the report.
This is the key part in CMSISDAPProtocol.dap_info:
https://github.com/pyocd/pyOCD/blob/e2796e50eacde5ca94aa3194204837e8b1f211b0/pyocd/probe/pydapaccess/cmsis_dap_core.py#L159-L168
Since the requested ID (id_) determines the string versus int response type, the only path where a string can be returned for the Capabilities request is if the value length in the response packet is something other than {1, 2, 4}. (Not shown here, None can be returned if the len is 0.) That would be a pretty corrupted USB packet.
I'm armour plating this code to better check the response. But you'll still get an exception if the response packet is invalid, so I'm not sure it will really help your situation.
(Btw, pyocd-flashtool is removed in version 0.30.0, as it has been deprecated for a couple years. Please switch to the combined pyocd tool with flash subcommand.)

authors: flit pfalcon

repo: pyocd/pyOCD


text:
Fix IFREMER URL and add SalishSeaCast to servers.

HI! I would like to add SalishSeaCast to the list of servers and I also corrected the IREMER's URL (it's shown to be incorrect here). Let me know if there is anything else I should do!

Thanks @circularpenguin!

authors: circularpenguin ocefpaf

repo: pyoceans/erddapy


text:
send update notifications on dry run

I would like to receive which containers have updates available through notifications using the "--dry-run" flag.
Currently it only sends the time it started looking for updates and when it will start the next time. The updates are only noticable through the logs.

Throwing my hat into the ring. I am interested also in getting notified of image updates when using --dry-run but I see where the problem lies with repeated notifications. Perhaps a new flag --notify-on-dry-run would suffice and the user can deal with repeated notifications until the images are upgraded.

I'm looking for a way to monitor for outdated containers, so I can update them within an update window (I will probably have to do this manually, without --dry-run-ning it). I'm currently monitoring everything through prometheus. This project can almost do what I want, except it should not do the actual update, but set a metric so prometheus can catch when an actual container needs an update.
Is this completely out of scope for this project? Is there a different way to monitor running containers and get notified when I have one that is outdated?

authors: eppixx michaelarnauts onedr0p

repo: pyouroboros/ouroboros


text:
Use arm64 MSVC on arm64 Windows

Related to https://github.com/pypa/setuptools/pull/4555 but with no dependency.

An alternative approach would be to conditionally construct PLAT_TO_VCVARS based on get_host_platform (which would then let you cross-compile to X86 or AMD64 using the native ARM64 compilers on ARM64).
The host platform should not be changing due to settings or anything else at runtime (the target platform theoretically should be able to be configured at runtime, though I'm not sure if it can).

It was really bothering me that we had two copies of essentially the same mapping, so I spent some time thinking about what is it that these two mappings represent and distilled from that a function that will construct the spec. This new function _get_vcvars_spec encodes the logic that was previously in comments only (that win32 supersedes everything but win-arm64). It additionally adds some doctests that perform some checks on the expected values. This logic should be more durable and maintainable than lookup tables.

diffcov tests are now failing for the changes in initialize, which is due to the fact that the coverage tests run on Linux and this project doesn't have coverage aggregation, so I'll ignore it for now.
Let me know if you have any lingering concerns; otherwise, it's ready to go.

Confirmed the behavior locally, LGTM

This logic should be more durable and maintainable than lookup tables.

It's a bigger compatibility change from what's currently there, and it seems possible (I won't say likely) that someone could be depending on that dict being there, either to read it or to update it.
Your call, of course, but if you're going to break compatibility then why not just implement it better in setuptools?

This logic should be more durable and maintainable than lookup tables.

It's a bigger compatibility change from what's currently there, and it seems possible (I won't say likely) that someone could be depending on that dict being there, either to read it or to update it.

My compatibility strategy for distutils is to first deprecate all such interactions with distutils (such that only setuptools is interacting with distutils), making distutils an implementation detail that Setuptools can then absorb (accept and run the tests) and converge.
The deprecation is in place, so there shouldn't be any users relying on this variable.
I know there's a possibility users are reliant on it and just haven't admitted such, but I'm willing to take that risk and provide a compatibility layer or back it out if necessary.

Why not just implement it better in setuptools? (Otherwise you'll never get rid of distutils ;) )

I see much of what Setuptools provides over distutils as smallish extensions of the core distutils functionality. I'm expecting that many (most?) of the interfaces in distutils will become the canonical implementation for Setuptools. I've been consolidating some behaviors and directing contributions here when relevant, rather than expanding the customizations in Setuptools.

authors: jaraco saschanaz zooba

repo: pypa/distutils