text:
Needs a PowerShell bootstrap script

It'd be nice if we could use ubi to install ubi, but that won't fly. So it'd be nice to provide a quick PowerShell bootstrap script instead.

Full disclosure: I am by no means an expert with PowerShell. I've only ever used it for a handful of tasks, and this may actually be the most I've used it for any single purpose to date.
That being said, I'm fairly confident this will work as I intended it to (even if I think it could be done better), and have partially tested it on my own Windows machine.
To start with, you can always find the page containing all downloads associated with the most recently uploaded release by navigating to https://github.com/houseabsolute/ubi/releases/latest/ (Which will redirect you to the release-specific URL). For interacting with arbitrary internet endpoints using PowerShell, the Invoke-WebRequest cmdlet is probably what you want.
From there, you may want to do some parsing to find what files are available to download for the release (In this case, I'm assuming the user isn't trying to download the source code, but the zip/tarball containing the pre-built binary), possibly query the user for which they want (I'm simply going to assume which file the user wants based on their platform), and then re-invoke Invoke-WebRequest using the URL for that specific file.
To determine the URL for the specific file download, take the URL the initial request redirects to, and replace the string '/tag/' with '/download/', then append a '/' followed by the name of the specific file to the end. Since your file name scheme doesn't include the version number, you won't need to extract that from the redirected URL, but I've included a commented line that will do that if you need it. As such, your final request will be to a URL of the form
https://github.com/houseabsolute/ubi/releases/download/VERSION/ubi-SYSTEM-x86_64.FILE_EXTENSION
For versions of PowerShell prior to PowerShell Core (version 6.0, I believe), there really isn't a great way to detect the host OS using exclusively PowerShell and cmdlets (that don't require newer functionality), but that shouldn't be a huge deal, since there isn't really a great way to (natively) run versions of PowerShell prior to 6.0 on non-Windows systems. Therefore, it's probably safe to just hard-code the download for Windows in that case. Otherwise, there are automatic variables defined according to the platform, which we can test for and accordingly use the correct replacement strings for SYSTEM and FILE_EXTENSION as used above.
The following code should work for at least versions of PowerShell >= 4 on any of the 3 mainstream 64-bit desktop operating systems, downloading the proper file from this repository's latest release and saving it to the current directory with the filename unchanged (At least when run from the PowerShell command line itself).
# Try our best to make sure we allow any security protocol we might reasonably need; admittedly, this is a security risk, but I leave exactly how to handle this up to you.
# This should remain applied only until the end of the PowerShell session unless subsequently modified.
# It might be prudent to remember the initial setting and re-apply it when finished.
[Net.ServicePointManager]::SecurityProtocol = 'Ssl3,Tls,Tls11,Tls12,Tls3'
## This second way is probably the "more correct" method, but its also far more verbose.
## It's probably not worth using it until/unless you come across a scenario where the above assignment isn't good enough.
## [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Ssl3 -bor [Net.SecurityProtocolType]::Tls -bor [Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12 -bor [Net.SecurityProtocolType]::Tls13

# Request to be redirected to a page with downloads for the latest release.
$DlFileURL = (Invoke-WebRequest -URI "https://github.com/houseabsolute/ubi/releases/latest/").BaseResponse.ResponseUri.OriginalString

## This is one way to get the version from this particular URL if necessary.
## $DlFileVer = Split-Path -leaf "$DlFileURL"

# Determine platform and corresponding file extension for download.
If ($IsLinux){ $TargetOS = "Linux"; $DlFileExtension = "tar.gz" }
ElseIf ($IsMacOS){ $TargetOS = "Darwin"; $DlFileExtension = "tar.gz" }
ElseIf ($IsWindows -or $PSVersionTable.PSVersion.Major -lt 6){ $TargetOS = "Windows"; $DlFileExtension = "zip" }

# Yes, this could simply be inlined within the above if statements, but there may eventually be a use for having just specific parts of the file name.
$DlFileName = "ubi-$TargetOS-x86_64.$DlFileExtension"

# Replace 'tag' in the URL we're initially redirected to with 'download' so that requesting the URL again will specifically target the latest version.
# The slashes on both sides are included in both strings just to be certain that only whole path segments are replaced.
$DlFileURL = "$DlFileURL" -replace '/tag/','/download/'

# Append the part of the URL that will specify exactly which file to download among those from this latest release.
$DlFileURL = "$DlFileURL/$DlFileName"

# Make the request to download the correct file.
Invoke-WebRequest -URI "$DlFileURL" -OutFile "$DlFileName"
# I'm not actually sure how you'd like to specify the location to place this file in, so I'll leave that for you to decide and implement.

What you do after that is a little bit beyond the scope of my limited knowledge, and I'm not currently available to do too much digging, but I think you might end up using the Invoke-Item cmdlet at some point to operate on the file you downloaded.
Older versions of Windows, like the version installed on my own machine, don't natively have a way to handle tar files (though that may not be a problem if you continue to use zip files for the windows releases), so it may be good to follow the second part of this answer from Stack Overflow. The same reasoning may also apply depending on the toolchain used during compilation, but I'm more focused on supporting older systems (and people who can't or won't update their OS).
If you don't want to deal with that, then you can probably use the Expand-Archive cmdlet to unzip the compressed binary for Windows targets, but you'll still have to figure out how you'd like to handle tar-zipped files for other targets (Perhaps using something like (Get-Command TAR_BINARY_NAME).Path to find whichever specific tar binary is standard each platform, then calling it with the correct parameters using Start-Process).
Of course, all of this presumably relies on the user having granted PowerShell permission to run scripts on their machine.
Hopefully I've been able to at least give you a starting point.

Full disclosure: I am by no means an expert with PowerShell. I've only ever used it for a handful of tasks, and this may actually be the most I've used it for any single purpose to date.
That being said, I'm fairly confident this will work as I intended it to (even if I think it could be done better), and have partially tested it on my own Windows machine.
To start with, you can always find the page containing all downloads associated with the most recently uploaded release by navigating to https://github.com/houseabsolute/ubi/releases/latest/ (Which will redirect you to the release-specific URL). For interacting with arbitrary internet endpoints using PowerShell, the Invoke-WebRequest cmdlet is probably what you want.
From there, you may want to do some parsing to find what files are available to download for the release (In this case, I'm assuming the user isn't trying to download the source code, but the zip/tarball containing the pre-built binary), possibly query the user for which they want (I'm simply going to assume which file the user wants based on their platform), and then re-invoke Invoke-WebRequest using the URL for that specific file.
To determine the URL for the specific file download, take the URL the initial request redirects to, and replace the string '/tag/' with '/download/', then append a '/' followed by the name of the specific file to the end. Since your file name scheme doesn't include the version number, you won't need to extract that from the redirected URL, but I've included a commented line that will do that if you need it. As such, your final request will be to a URL of the form
https://github.com/houseabsolute/ubi/releases/download/VERSION/ubi-SYSTEM-x86_64.FILE_EXTENSION
For versions of PowerShell prior to PowerShell Core (version 6.0, I believe), there really isn't a great way to detect the host OS using exclusively PowerShell and cmdlets (that don't require newer functionality), but that shouldn't be a huge deal, since there isn't really a great way to (natively) run versions of PowerShell prior to 6.0 on non-Windows systems. Therefore, it's probably safe to just hard-code the download for Windows in that case. Otherwise, there are automatic variables defined according to the platform, which we can test for and accordingly use the correct replacement strings for SYSTEM and FILE_EXTENSION as used above.
The following code should work for at least versions of PowerShell >= 4 on any of the 3 mainstream 64-bit desktop operating systems, downloading the proper file from this repository's latest release and saving it to the current directory with the filename unchanged (At least when run from the PowerShell command line itself).
# Try our best to make sure we allow any security protocol we might reasonably need; admittedly, this is a security risk, but I leave exactly how to handle this up to you.
# This should remain applied only until the end of the PowerShell session unless subsequently modified.
# It might be prudent to remember the initial setting and re-apply it when finished.
[Net.ServicePointManager]::SecurityProtocol = 'Ssl3,Tls,Tls11,Tls12,Tls3'
## This second way is probably the "more correct" method, but its also far more verbose.
## It's probably not worth using it until/unless you come across a scenario where the above assignment isn't good enough.
## [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Ssl3 -bor [Net.SecurityProtocolType]::Tls -bor [Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12 -bor [Net.SecurityProtocolType]::Tls13

# Request to be redirected to a page with downloads for the latest release.
$DlFileURL = (Invoke-WebRequest -URI "https://github.com/houseabsolute/ubi/releases/latest/").BaseResponse.ResponseUri.OriginalString

## This is one way to get the version from this particular URL if necessary.
## $DlFileVer = Split-Path -leaf "$DlFileURL"

# Determine platform and corresponding file extension for download.
If ($IsLinux){ $TargetOS = "Linux"; $DlFileExtension = "tar.gz" }
ElseIf ($IsMacOS){ $TargetOS = "Darwin"; $DlFileExtension = "tar.gz" }
ElseIf ($IsWindows -or $PSVersionTable.PSVersion.Major -lt 6){ $TargetOS = "Windows"; $DlFileExtension = "zip" }

# Yes, this could simply be inlined within the above if statements, but there may eventually be a use for having just specific parts of the file name.
$DlFileName = "ubi-$TargetOS-x86_64.$DlFileExtension"

# Replace 'tag' in the URL we're initially redirected to with 'download' so that requesting the URL again will specifically target the latest version.
# The slashes on both sides are included in both strings just to be certain that only whole path segments are replaced.
$DlFileURL = "$DlFileURL" -replace '/tag/','/download/'

# Append the part of the URL that will specify exactly which file to download among those from this latest release.
$DlFileURL = "$DlFileURL/$DlFileName"

# Make the request to download the correct file.
Invoke-WebRequest -URI "$DlFileURL" -OutFile "$DlFileName"
# I'm not actually sure how you'd like to specify the location to place this file in, so I'll leave that for you to decide and implement.

What you do after that is a little bit beyond the scope of my limited knowledge, and I'm not currently available to do too much digging, but I think you might end up using the Invoke-Item cmdlet at some point to operate on the file you downloaded.
Older versions of Windows, like the version installed on my own machine, don't natively have a way to handle tar files (though that may not be a problem if you continue to use zip files for the windows releases), so it may be good to follow the second part of this answer from Stack Overflow. The same reasoning may also apply depending on the toolchain used during compilation, but I'm more focused on supporting older systems (and people who can't or won't update their OS).
If you don't want to deal with that, then you can probably use the Expand-Archive cmdlet to unzip the compressed binary for Windows targets, but you'll still have to figure out how you'd like to handle tar-zipped files for other targets (Perhaps using something like (Get-Command TAR_BINARY_NAME).Path to find whichever specific tar binary is standard each platform, then calling it with the correct parameters using Start-Process).
Of course, all of this presumably relies on the user having granted PowerShell permission to run scripts on their machine.
Hopefully I've been able to at least give you a starting point.

Thanks very much for this!
I trimmed it down to something a bit simpler since I already have a shell script for everything that's not Windows. You can see the final result at https://github.com/houseabsolute/ubi/blob/master/bootstrap/bootstrap-ubi.ps1

Thanks very much for this!
I trimmed it down to something a bit simpler since I already have a shell script for everything that's not Windows. You can see the final result at https://github.com/houseabsolute/ubi/blob/master/bootstrap/bootstrap-ubi.ps1

Glad I could help!

Glad I could help!

authors: StaticPH autarch

repo: houseabsolute/ubi


text:
Slack RTM retry/reconnect not working

Hi,
I've found that the RTM reconnect logic isn't reached when the RTM connection dies for one reason or another.
Version: 0.1.2
OS: Ubuntu 15.04
Node: v6.1.0
To test I have done the following:

Set retry to 5 in my controller config
console.log'd the retryEnabled var in Slackbot_worker.js to ensure it was set
console.log'd at the start of reconnect()
tcpkill'd the RTM connection
Confirmed nothing was logged to the console

This follows a series of less drastic actions to confirm it wasn't working.
It seems the only path to the retry logic is at the bottom of bot.closeRTM when an err object is passed. This code is not reached from the error or close event of bot.rtm which is the path followed when the connection dies unexpectedly. I am unsure whether or not it is desirable to call reconnect() from close, so I've not submitted a PR. I'm happy to once it's been clarified what would be desired. As a user I would probably expect the retry logic to occur on a team_migration_started event also, though I suspect that would be be immediately followed by a close event as per the Slack documentation.
Thank you for botkit, it's great stuff!
Cheers,
Gary.

Hi, I have been having issues with my slack connection just going dark as well and with your suggestion of tcpkilling the connection, I found a few issues that I tried to fix here:
https://github.com/howdyai/botkit/pull/298
I'm not sure if it addresses your specific issue or not, but once I was logging properly (https://github.com/howdyai/botkit/pull/299), I was able to see slackbot do reconnection attempts with tcpkill.

Ah, this looks like the issue I'm having upon hitting a debug breakpoint. I don't know enough about @jayjanssen's PR but it definitely touches what I'm seeing go wrong with reconnects.

I'm using v0.4 that includes the fix above, but also found this in logs:
error: Error: Stale RTM connection, closing RTM
After this no messages reach slack and the bot is offline, so the issue is still there.

For anyone looking for a reliable solution, I have auto-reconnect disabled and use the following code to keep the bot alive. It hasn't failed once since my last reply here.

var controller = ...;
var bot = ...;

function start_rtm() {
        bot.startRTM(function(err,bot,payload) {
                if (err) {
                        console.log('Failed to start RTM')
                        return setTimeout(start_rtm, 60000);
                }
                console.log("RTM started!");
                });
        });
}

controller.on('rtm_close', function(bot, err) {
        start_rtm();
});

start_rtm();

Hope it helps!

@garymoon How do I disable auto-reconnect? bot.config.retry = false?

@rickul It's disabled by default, I just meant I don't have it enabled when I use the code above.

Thanks! Your code works fine 👍

Closed as answered

Probably best to remove the feature if there's no interest in fixing it @peterswimm.

@garymoon with the shift from rtm to events you may see a deprecation of this feature, but if anyone has a PR that improves functionality of auto-reconnect we will definitely consider it.

@peterswimm My understanding is that events and RTM are intended to compliment one another, indeed each has several features not supported by the other.
It seems that silent disconnects are causing some confusion for those using RTM with botkit (myself included). With some guidance on what would be expected of a PR to resolve this issue (assuming transplanting my solution above into the library itself is undesirable) I believe I'd be able to submit a working PR.

I tried @garymoon code and I went to a issue, at some points it was start twice, and two websockets were opened. I didn't enable "retry", looks like it goes to this code
`bot.rtm.on('close', function(code, message) {
botkit.log.notice('RTM close event: ' + code + ' : ' + message);
if (pingIntervalId) {
clearInterval(pingIntervalId);
}
botkit.trigger('rtm_close', [bot]);
            /**
             * CLOSE_ABNORMAL error
             * wasn't closed explicitly, should attempt to reconnect
             */
            if (code === 1006) {
                botkit.log.error('Abnormal websocket close event, attempting to reconnect');
                reconnect();
            }
        });`

it calls reconnect function without checking retry and @garymoon called the same thing rtm_close. not sure if I'm correct and how to fix this one

@xuanvinh2005 After struggling with this, I found this bug and created this PR
https://github.com/howdyai/botkit/pull/532
Very simple change from bot to botkit. Reconnect has bee working flawlessly since.  Also using the following code to notify me in slack of reconnects, drops, and other slack RTM messages.
`controller.on('rtm_close',function(bot) {
console.log('\n\n*** '+moment().tz('America/Los_Angeles').format()+'  ** The RTM api just closed');
try {
controller.findTeamById('<SLACK_TEAM_ID_HERE>', function(err, team) {
var alertBot = controller.spawn(team);
alertBot.api.chat.postMessage(
{
channel : '<SLACK_CHANNEL_ID_HERE>',
text: 'GUYS!!! I just got an RTM Close event trigger for this team --> '+bot.team_info.id+'  '+bot.team_info.name+' .',
attachments: [],
}, function (err, response) {
});
});
} catch (err) {
console.log('\n\n*** '+moment().tz('America/Los_Angeles').format()+' Tried to send alert for an rtm_close event but failed with this error: '+err);
}
});
controller.on('rtm_reconnect_failed',function(bot) {
console.log('\n\n*** '+moment().tz('America/Los_Angeles').format()+'  ** Unable to automatically reconnect to rtm after a closed conection.');
try {
controller.findTeamById('<SLACK_TEAM_ID_HERE>', function(err, team) {
var alertBot = controller.spawn(team);
alertBot.api.chat.postMessage(
{
channel : '<SLACK_CHANNEL_ID_HERE>',
text: 'GUYS!!! I just got an RTM Reconnect Failed trigger for this team --> '+bot.team_info.id+'  name: '+bot.team_info.name+'. The team either uninstalled the app or there was an issue which requires a resart of the bot.',
attachments: [],
}, function (err, response) {
});
});
} catch (err) {
console.log('\n\n*** '+moment().tz('America/Los_Angeles').format()+' Tried to send alert for an rtm_reconnect_failed event but failed with this error: '+err);
}
});
`

About retry stuff, I have workaround by setting retry when calling spawn instead of in botkit config,
var bot = controller.spawn({             token: token, 	    retry: 10         });
however, my issue is the same with @rdohms's error: Error: Stale RTM connection, closing RTM
bot was died without reconnect, I tried to increase ping/pong check time from 1200 to bigger value, it seems to be fine

Had the same issue but once I fixed the bug in Slackbot_worker.js it has been working fine and haven't had any issues in the last month.
Just to make sure, I logged the status of the retryEnabled attribute in the closeRTM function in Slackbot_worker.js. With the above change, it was always true for the stale connection error and haven't had any issues since. Hope this helps.
    bot.closeRTM = function(err) {
      console.log('\n\n **** THE closeRTM FUNCTION WAS JUST CALLED AND THIS IS THE ERROR --> '+err+' AND RETRY IS --> '+retryEnabled);
        if (bot.rtm) {
            bot.rtm.removeAllListeners();
            bot.rtm.close();
        }

        if (pingIntervalId) {
            clearInterval(pingIntervalId);
        }

        lastPong = 0;
        botkit.trigger('rtm_close', [bot, err]);

        // only retry, if enabled, when there was an error
        if (err && retryEnabled) {
            reconnect();
        }
    };

source: gharchive/issue

authors: DustinVenegas garymoon inn0v8 jayjanssen peterswimm rdohms rickul xuanvinh2005

repo: howdyai/botkit

license: mit

license_type: permissive

license_source: bigquery


text:
Config values are not saved

Describe the bug
Toggling any of the four parameters in plugin settings, makes no difference (Settings are not saved)

To Reproduce
Steps to reproduce the behavior:

Go to Plugins
Click on Settings for Nibe Uplink
Scroll down to Accessories
Try to toggle one of them, and save, restart. Values are not saved

Expected behavior
I would have expected the ability to turn off some accessories via config
Screenshots & Logfiles
See above
Versions
Plugin version: 0.1.2
Node version: Not sure
Operating system: Inside homebridge docker

Value was not in schema so Google checkbox was updating config but save button was validating it with schema and removing not existing properties.
I will release it today.

v0.1.3 released. @robinjhector please let me know if it works after update or just simply close this issue.

authors: hpruszyn robinjhector

repo: hp-net/homebridge-nibe

license: MIT

license_source: github-api


text:
[tests] diffuser models in model zoo

📌 Checklist before creating the PR

[ ] I have created an issue for this PR for traceability
[ ] The title follows the standard format: [doc/gemini/tensor/...]: A concise description
[ ] I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge
e.g. fixed #1234, closed #1234, resolved #1234

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

I added diffuser models in model zoo.
💥 Checklist before requesting a review

[ ] I have linked my PR to an issue (instruction)
[ ] My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
[x] I have performed a self-review of my code
[x] I have added thorough tests.
[ ] I have added docstrings for all the functions/methods I implemented

⭐ Do you enjoy contributing to Colossal-AI?

[x] 🌝 Yes, I do.
[ ] 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

Just wanna mark that this PR is part of efforts for #3049

source: gharchive/pull-request

repo: hpcaitech/ColossalAI


text:
agent_sim: Fix order of parameters

When calling agent_sim_process(), the order of the parameters was wrong.

why this instead of https://github.com/hpcloud-mon/monasca-perf/blob/master/scale_perf/agent_simulator.py

why this instead of https://github.com/hpcloud-mon/monasca-perf/blob/master/scale_perf/agent_simulator.py

@mhoppal I didn't saw that script. I'll have a look

@mhoppal hm. should we delete the agent_sim script completly?

probably unless you see any use out of it? We really havent done a good job keeping only the scripts we use.

I don't plan to work on this. Closing.

authors: mhoppal toabctl

repo: hpcloud-mon/monasca-perf


text:
[EXTERNAL] Reservando recursos...Malloc - ERROR! Please help

I have test dpdk2disk with default version dpdk inlude this. but get error message like this.Pleas help this issue.
root@knot-onesys:/home/dhb/DPDK2disk# ./scripts/capture0.sh /home/dhb/
EAL: Detected 8 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:02:01.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:02:02.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:02:03.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10fb net_ixgbe
[EXTERNAL] Se define un WSlave en el Core 4
[EXTERNAL] Carpeta establecida a /home/dhb/.
[EXTERNAL] Tamaño maximo de fichero establecido a 4294967296 bytes
Creating the mbuf pool for socket 0 ...
Creating ring to connect I/O lcore 1 (socket 0) with worker lcore 3 ...
Creating ring to connect worker lcore 3 with TX port 0 (through I/O lcore 2) (socket 0) ...
Initializing NIC port 0 ...
Initializing NIC port 0 RX queue 0 ...
Initializing NIC port 0 TX queue 0 ...
PMD: ixgbe_dev_link_status_print():  Port 0: Link Down
Checking link status.................................done
Port 0 Link Up - speed 1000 Mbps - full-duplex
[EXTERNAL] Reservando recursos...Malloc - ERROR!
/home/dhb/DPDK2disk
root@knot-onesys:/home/dhb/DPDK2disk#

Hi,
First of all, sorry for that "in spanish" error message.
The problem is that this app uses LOTS of ram, see https://github.com/hpcn-uam/DPDK2disk/blob/master/src/external.c at line 162.
It is trying to allocate sizeof(BUFFERKIND)diskBufferSize witch means 1102410241024 bytes (1GiB).
Show me your hugepage setup.
Also I do not recommend to use /home as output directory, you probably want to use some kind of raid 0 or NVMe array, specially if you want to capture >5G or so.

Hi  ralequi:
That is very kind of you to help me , as I was new to test DPDK, and my test lab use Vmware Esxi 6.5 run ubuntu 16.04 with Vxnet3 and two intel X520(pci passthrougth).I have other test server with SSD install but not now. I alway get DPDK errror with VM. and I may not repeat this issue. Here is steps I try:
1.check DPDK version(seems the same with DPDK2disk git)
root@knot-onesys:/home/dhb/DPDK2disk/dpdk# pwd
/home/dhb/DPDK2disk/dpdk
root@knot-onesys:/home/dhb/DPDK2disk/dpdk# git branch

(HEAD detached at d84aad6)
master
root@knot-onesys:/home/dhb/DPDK2disk/dpdk#
2.try set up DPDK
./usertools/dpdk-setup.sh
Configuration done using x86_64-native-linuxapp-gcc
== Build lib
== Build lib/librte_compat
== Build lib/librte_eal
== Build lib/librte_eal/common
== Build lib/librte_eal/linuxapp
== Build lib/librte_eal/linuxapp/eal
== Build lib/librte_eal/linuxapp/igb_uio
........

set hugepages
Option: 21

Removing currently reserved hugepages
Unmounting /mnt/huge and removing directory
Input the number of 2048kB hugepages for each node
Example: to have 128MB of hugepages available per node in a 2MB huge page system,
enter '64' to reserve 64 * 2MB pages on each node
Number of pages for node0: 2048
Reserving hugepages
Creating /mnt/huge and mounting as hugetlbfs
Press enter to continue ...
4.check hugepages
cat /proc/meminfo | grep Huge
AnonHugePages:      8192 kB
HugePages_Total:    2048
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
5.check interface
dpdk_nic_bind --status
Network devices using DPDK-compatible driver
0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe
0000:0b:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe
6.try test capture
./scripts/capture0.sh /home/dhb/
I get error message :
EAL: Detected 8 lcore(s)
EAL: lcore 8 unavailable
EAL: invalid coremask
Here is my cpu layout:
./dpdk/usertools/cpu_layout.py
Core and Socket Information (as reported by '/sys/devices/system/cpu')
cores =  [0, 1, 2, 3, 4, 5, 6, 7]
sockets =  [0]
   Socket 0
   --------

Core 0 [0]
Core 1 [1]
Core 2 [2]
Core 3 [3]
Core 4 [4]
Core 5 [5]
Core 6 [6]
Core 7 [7]
I have "vi ./scripts/capture0.sh" change "sudo build/app/hpcn_n2d -c FFF" to "sudo build/app/hpcn_n2d -c 0xff -n 6"  (I am not sure if this is right?)

try ./scripts/capture0.sh /home/dhb/

get error messge:
EAL: Detected 8 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:13:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15ad:7b0 net_vmxnet3
EAL: PCI device 0000:1b:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15ad:7b0 net_vmxnet3
[EXTERNAL] Se define un WSlave en el Core 4
[EXTERNAL] Carpeta establecida a /home/dhb/.
[EXTERNAL] Tamaño maximo de fichero establecido a 4294967296 bytes
Creating the mbuf pool for socket 0 ...
Creating ring to connect I/O lcore 1 (socket 0) with worker lcore 3 ...
Creating ring to connect worker lcore 3 with TX port 0 (through I/O lcore 2) (socket 0) ...
Initializing NIC port 0 ...
Initializing NIC port 0 RX queue 0 ...
Initializing NIC port 0 TX queue 0 ...
PMD: ixgbe_dev_link_status_print():  Port 0: Link Down
Checking link status..........................done
Port 0 Link Up - speed 1000 Mbps - full-duplex
[EXTERNAL] Reservando recursos...Malloc - ERROR!
/home/dhb/DPDK2disk
Thank you for your help.

Hi,
Sorry to take too long on response. I absolutely forget about it :-(. (too much work to remember everything u probably know...)

First of all. VMs have some degradation on DPDK. It's not like a terrible thing but it is present. Read the results present here: https://www.dit.upm.es/~jlopez/publicaciones/comnet19.pdf

Second, SSD can be nice for some initial tests, but a single ssd can't handle network capture. If I remember correctly, you need at least 4 SSD to capture at 10gbit, but even with that, it is not posible to keep it capturing for long time, since you would have SSD degradation, witch it's only fix is to perform a "secure-erase". Keep that in mind when using SSDs.

Third, DPDK recomends 1G hugepages. Insert this options to your kernel at boot: hugepagesz=1G default_hugepagesz=1G hugepages=16. You should put that on host and VMs. Also, vmware should be aware and force to allocate VM memory over hugepages to get the best performance with any DPDK app.

4º, the scripts stored at /scripts, are ad-hoc. With that said, open it and change whatever you want, in this case, the coremask (-c option to the underlay application). In this case -c 0xff (As you have 8 cores) may be an option. Remember that the smallest core in the mask is going to be used by dpdk internally. Answering to your "7º", that mask is correct for your setup.

5º, I'm not 100% sure, but the memory problem may come from using 2MB pages and low number of pages. You have allocated 4GB, (which in fact is exactly the memory required for that malloc, you need some extra mem for mbufs, etc). Also the rte_malloc functions requires to have the memory "physically contiguous" (not virtual), a fact that is not "as easy as you may think" to achieve with 2MB pages, specially if the host is running for more than a couple of hours or have some heavy mem processes (such a graphic interface).

authors: missyoyo ralequi


text:
Show or hide

Hello,
when I use show-if-fullscreen directive, it only works as expected for the first time. Second time the icons don't change properly (or at all).
Here is my code:
<md-button ngsf-toggle-fullscreen show-if-fullscreen-enabled hide-xs class="md-icon-button"> <md-icon show-if-fullscreen=false md-svg-icon="fullscreen"></md-icon> <md-icon show-if-fullscreen=true md-svg-icon="fullscreen_exit"></md-icon> </md-button>

Can you create a plunkr with the code you are trying? Or tell me if mine works.
Here is the example I use, http://plnkr.co/edit/akbNXmHGvPqHydqXot1w?p=preview
In order for it to work you have to click on the "Launch the preview in a separated window" icon on the top right of the preview as the fullscreen doesn't work on iframes

Closed for now, if issue arises again reopen with the code you are trying

authors: filipSu hrajchert

repo: hrajchert/angular-screenfull


text:
Is the intention of this different from NASA-UTM 'put-operation'?

https://app.swaggerhub.com/apis/utm/uss/v4#/A.%20Required%20Endpoints/put_enhanced_operations__gufi_
Is this redundant? Or are the two trying to solve a different problem? Apologies for maybe coming in late into this discussion.

Also I guess the flight declaration protocol was developed internally at GUTMA and this repository takes that work further. I am happy to discuss with you offline as well to bridge this gap between the two.

authors: hrishiballal overlandla


text:
Escape dollars in names to prevent env-var expansion

Dollars in filenames need to be escaped so they aren't interpreted as variables for expansion

Having $ in filenames is something that is common in Remix (dynamic segments)

Better to escape other special chars, e.g. %, \ as well using fnameescape(), see https://github.com/hrsh7th/cmp-cmdline/pull/122

authors: Bekaboo joshbode


text:
Use lua in after plugin

Use lua in after plugin
This seems to solve an issue I have on the first install using packer. Other cmp plugins didn't have this error and this seems to be the only difference. I imagine the vim calling lua to require cmp happens too late for packer when it runs? Not sure.
async.lua:20: Error in coroutine: ...ack/packer/start/packer.nvim/lua/packer/plugin_utils.lua:204: Vim(lua):E5108: Error executing lua [string ":lua"]:1: module 'cmp' not found:
no field package.preload['cmp']

Looks good. Thank you!


text:
Is there a way to add border to the suggestions/buffer floating window?

FAQ

[X] I have checked the FAQ and it didn't resolve my problem.

Issues

[X] I have checked existing issues and there are no open or closed issues with the same problem.

Neovim Version
NVIM v0.7.0-dev+665-g4e352ecc4
Minimal reproducible config
Question, NOT A BUG.
Description
Hi, thank you for this excellent plugin.
Is there a way to add border to the suggestions/buffer floating window? I know I can add border to the documentation window, but it's not the same.
Thank you
Steps to reproduce
N/A
Expected behavior
N/A
Actual behavior
N/A
Additional context
N/A

No, please see #472

perfect, looks like there is a PR in the making, near completion. Thank you.


text:
Update Travis white list

Fixing Travis issues

Hm temporary issue? Feel free to close

@ReadmeCritic thanks for taking the time to investigate this. i can't figure out what's wrong either


text:
较多更新,增加一些通用功能,如 "热替换,mock,oidc认证, 调试组件"等

1.增加开发插件 customize-cra-react-refresh, 使用 热替换而不是热重载
2. 添加 mock MSW 即 mock service work 模拟服务请求特性
3. 用户登录移入 mock
4. 增强部分ts类型, 并引入一些typescript类型
5. 增加devServer配置选项, 并加入更灵活的proxy配置 ,并修改不可用的proxy配置
6. 修改axios响应转换,使得更符合一般的情况. 把"网络异常" 改为 "请求失败"避免歧义
7. 增加 oidc登录,和原功能并行
8. 增加.env配置文件, 配置oidc信息
9. 调整 package.json 的 依赖和开发依赖错误位置 及一些需要的库如 oidc_client, redux-devtools
10. 增加Json查看组件 ,用于调试
11. 其他修改 和 代码格式自动修改( 有可能是因为编辑器或者prettier设置问题导致的 :(  )

@tearf001 好的，非常感谢。我先review看看😉

抱歉,这里面较多修改是没必要代码格式化变化引起的.
可能是注释了 'prettier/prettier': ['error', {parser: 'flow'}] ,(这句在webstorm上有bug, ..时间有点仓促没有研究)
后期再增加一些约定...

Prettier 官网 关于 Parser 的文档
Specify which parser to use.
Prettier automatically infers the parser from the input file path, so you shouldn’t have to change this setting.
Both the babel and flow parsers support the same set of JavaScript features (including Flow type annotations). They might differ in some edge cases, so if you run into one of those you can try flow instead of babel. Almost the same applies to typescript and babel-ts. babel-ts might support JavaScript features (proposals) not yet supported by TypeScript, but it’s less permissive when it comes to invalid code and less battle-tested than the typescript parser.
Valid options:

"babel" (via @babel/parser) Named "babylon" until v1.16.0
"babel-flow" (same as "babel" but enables Flow parsing explicitly to avoid ambiguity) First available in v1.16.0
"babel-ts" (similar to "typescript" but uses Babel and its TypeScript plugin) First available in v2.0.0
"flow" (via flow-parser)
"typescript" (via @typescript-eslint/typescript-estree) First available in v1.4.0
"espree" (via espree) First available in v2.2.0
"meriyah" (via meriyah) First available in v2.2.0
"acorn" (via acorn) First available in v2.6.0
"css" (via postcss-scss and postcss-less, autodetects which to use) First available in v1.7.1
"scss" (same parsers as "css", prefers postcss-scss) First available in v1.7.1
"less" (same parsers as "css", prefers postcss-less) First available in v1.7.1
"json" (via @babel/parser parseExpression) First available in v1.5.0
"json5" (same parser as "json", but outputs as json5) First available in v1.13.0
"json-stringify" (same parser as "json", but outputs like JSON.stringify) First available in v1.13.0
"graphql" (via graphql/language) First available in v1.5.0
"markdown" (via remark-parse) First available in v1.8.0
"mdx" (via remark-parse and @mdx-js/mdx) First available in v1.15.0
"html" (via angular-html-parser) First available in 1.15.0
"vue" (same parser as "html", but also formats vue-specific syntax) First available in 1.10.0
"angular" (same parser as "html", but also formats angular-specific syntax via angular-estree-parser) First available in 1.15.0
"lwc" (same parser as "html", but also formats LWC-specific syntax for unquoted template attributes) First available in 1.17.0
"yaml" (via yaml and yaml-unist-parser) First available in 1.14.0

下周再验证一下, 选择一个正确的parser

我把 parser 的问题重新审视了一下, 发现对代码格式化并无影响.
我想代码格式化问题 出在你的开发环境.
比如 prettier的配置 semi: false;  原代码中没有截取.

应统一 prettier的解析器为 typescript. 如 prettierrc中的配置
原.eslintrc 配置为 flow.
我已经按最新的  PR 了

升级 redux 原版 到 redux/toolkit, 减少大量模板代码,比如 actions, connect等
代码更容易跟踪和阅读, 具体来说:

1 不再需要 connect

- export default connect((state) => state, actions)(Header)
+ export default Header

2  props 不再需要扩展ReduxProps , 属性只保留组件相关

- interface Props - extends ReduxProps {
+ interface Props {
  option: object
  style?: object
}

3 不再需要声明 actions魔法字符串 , reducer 不再需要 switch action.type, 并把分离的 reducer / state / actionCreator 放入同一个 redux 切片 slice

- @/store/actrions
- @/store/reducers
- ...
+ @/store/slicers

slice 申明示例
export const tabSlice = createSlice({
  name: 'tab',
  initialState,
  // The `reducers` field lets us define reducers and generate associated actions
  reducers: {
    setTabs: (state, action: PayloadAction<any[]>) => {
      state.curTab = action.payload
    },
    setCurrentTab: (state, action) => {
      state.currentTab = action.payload
    },
    setReloadPath: (state, action) => {
      state.reloadPath = action.payload
    }
  },
  // The `extraReducers` field lets the slice handle actions defined elsewhere,
  // including actions generated by createAsyncThunk or in other slices.
  extraReducers: (builder) => {
    builder // 和其他 reducer互动, 或者自定义的 `AsyncThunk` Action
      .addCase(demoAsync.fulfilled, (state, action) => {
        state.status = 'idle'
        state.currentTab = action.payload
      })
  }
})
// actions
export const { setTabs, setCurrentTab, setReloadPath } = tabSlice.actions
// 强类型选择器
export const selectTabs = (state: RootState) => state.tab.curTab
export const selectCurrentTab = (state: RootState) => state.tab.currentTab
export const selectReloadPath = (state: RootState) => state.tab.reloadPath

4 状态获取和行为派发更加清晰可见, 并增强 redux state 类型提示

+ const dispatch = useAppDispatch()
+ const curTab = useAppSelector(selectTabs) // selectXxx 从 xxxSlice中导出
+ const reloadPath = useAppSelector(selectReloadPath) // selectXxx 从 xxxSlice中导出
... // 派发
+ dispatch(setReloadPath(pathname + search)) // setXxx 也从xxxSlice中导出

@tearf001 很奇怪，我把你的分支合并过来了，但是 【 Contributors 】 怎么没有你头像？贡献者应该都会展示啊！

@tearf001 刚刚合并代码后，我本地跑了一下，权限测试页面会报错，找不到 userInfo对象，你会这样吗

@tearf001 你确实是漏改了权限测试页的，selectUserInfo 和 setUserInfo，导致报错了。我改提交了


text:
fix dynamodb write throttling

Issue

DynamoDB bulding_approvals table throtles when putting items

Proposed Solutions

Redesign table to improve performance - distribute loads more evenly by changing keys

#11


text:
Missing easier way to return text/html content

Looking at the 'graphql' example, I have to depend on the mime crate to set text/html content-type:
let res = Response::new(StatusCode::Ok)
        .body_string(body)
        .set_mime(mime::TEXT_HTML_UTF_8);

if I don't want to depend on the mime crate, I can do:
    let res = Response::new(StatusCode::Ok)
        .body_string(body)
        .set_mime("text/html".parse().unwrap());

This looks non-optimal to me.
Such a basic use-case should be easier. I am wondering if I am missing an alternative.

On a related note, why is .set_mime() consuming the response? Shouldn't a mutable reference suffice and make the method more versatile?

I would personally prefer something like body_html or body_html_stirng or body_html since you most likely always want to pass charset i.e. text/html; charset=UTF-8.

Yeah, body_html/body_html_string would make good additions.

Created a small PR with this suggestion. Please consider it.

I'm curious how you're getting a string of html into your tide app. Is it being generated by a template engine? (if so, which one?) Or included through the include_bytes! macro? Or are you directly typing html in your route handler?

I used a template engine, handlebars.

I use ructe.
With the latest v0.10.0 release we don't require mime crate and can directly use tide::http::mime.
pub async fn not_found(_req: Request<AppState>) -> tide::Result {
    let mut res = Response::new(StatusCode::NotFound);
    res.render_html(|o| Ok(templates::notfound(o)?))?;
    Ok(res)
}

Given that we can't chain with v0.10.0 :(, I instead went ahead and created a trait that extends Response to include render  and render_html .
pub trait Render {
    fn render<Call>(&mut self, call: Call) -> std::io::Result<()>
    where
        Call: FnOnce(&mut dyn std::io::Write) -> std::io::Result<()>;

    fn render_html<Call>(&mut self, call: Call) -> std::io::Result<()>
    where
        Call: FnOnce(&mut dyn std::io::Write) -> std::io::Result<()>;
}

impl Render for tide::Response {
    fn render<Call>(&mut self, call: Call) -> std::io::Result<()>
    where
        Call: FnOnce(&mut dyn std::io::Write) -> std::io::Result<()>,
    {
        let mut buf = vec![];
        call(&mut buf)?;
        self.set_body(buf);
        Ok(())
    }

    fn render_html<Call>(&mut self, call: Call) -> std::io::Result<()>
    where
        Call: FnOnce(&mut dyn std::io::Write) -> std::io::Result<()>,
    {
        self.render(call)?;
        self.set_content_type(tide::http::mime::HTML);
        Ok(())
    }
}

authors: gmosx jbr prabirshrestha

repo: http-rs/tide


text:
docs: Add missing serde requirement in example

Hello :)
First of all, thank you for creating and maintaining this crate. I enjoy it :)
This PR fixes the documentation for the "Getting started" example inside lib.rs. Without serde in dependencies, the compiler throws the following error:
error[E0463]: can't find crate for `serde`
 --> src/main.rs:5:17
  |
5 | #[derive(Debug, Deserialize)]
  |                 ^^^^^^^^^^^ can't find crate
  |
  = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

I noted that the similar example in README has serde in place and copied it here :)

Thanks for catching this!

Thanks for catching this!


text:
Adds fasthttp handler

This adds the FastHttp implementation which is needed by dapr.
Depends on #4

rebased to depend on #4 as that centralizes the ABI changes

repo: http-wasm/http-wasm-host-go


text:
🛑 Manhuagui is down

In aee8e94, Manhuagui (https://www.manhuagui.com) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Manhuagui is back up in 49abdb0 after 6 minutes.


text:
Improve startup time when pyOpenSSL is available on the environment

Based on #1232

Codecov Report

Merging #1233 (0fd307d) into master (4d7d6b6) will decrease coverage by 0.58%.
The diff coverage is 94.81%.

@@            Coverage Diff             @@
##           master    #1233      +/-   ##
==========================================
- Coverage   97.28%   96.69%   -0.59%
==========================================
  Files          67       81      +14
  Lines        4235     5387    +1152
==========================================
+ Hits         4120     5209    +1089
- Misses        115      178      +63

Impacted Files
Coverage Δ

tests/test_binary.py
100.00% <ø> (ø)

httpie/compat.py
31.11% <27.90%> (-68.89%)
:arrow_down:

tests/conftest.py
77.14% <58.33%> (-9.82%)
:arrow_down:

tests/test_ssl.py
91.01% <66.66%> (-3.93%)
:arrow_down:

httpie/manager/__main__.py
82.35% <82.35%> (ø)

httpie/models.py
94.73% <83.33%> (-2.64%)
:arrow_down:

httpie/uploads.py
98.68% <83.33%> (-1.32%)
:arrow_down:

httpie/output/formatters/colors.py
92.66% <88.88%> (-0.92%)
:arrow_down:

httpie/manager/core.py
92.85% <92.85%> (ø)

httpie/manager/plugins.py
93.39% <93.39%> (ø)

... and 53 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df58ec6...0fd307d. Read the comment docs.

authors: codecov-commenter isidentical

repo: httpie/httpie


text:
why Inherit boomer

I wanna know whether it's more stable or strong performance.is there some data comparison
Best regards

@Winspain The HttpRunner+'s load generator part will rely on boomer, thus the performance should be similar to boomer.

authors: Winspain debugtalk

repo: httprunner/hrp


text:
🛑 Oldmai WEBUI is down

In 7b66711, Oldmai WEBUI (https://my-aime.cn) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Oldmai WEBUI is back up in b134f49 after 5 minutes.

repo: huajip/uptime


text:
Scheme 类中能添加一个成员变量为Object类型的吗，现在只有int和String无法满足我们的需求

Scheme 类中能添加一个成员变量为Object类型的吗，现在只有int和String无法满足我们的需求。

很快就有了，最近忙，没有发版本，很多可以改进的

感谢大神回复，期待更新

6666

authors: cuixbo hfkai

repo: huanghaibin-dev/CalendarView


text:
Mccalluc/special hubmap restrictions

I think we'll want to pin the version of the HCA schemas that we accept: The code downstream will be looking for particular fields in particular places. Since I think we'll be creating the JSON in the first place, this shouldn't be a huge impediment.
Fix #4 (though I expect this to grow.)

@shirey : I've made a branch off of this to continue my work; Please review, or let me know if you see problems.

repo: hubmapconsortium/hubmap-sample-metadata


text:
[Feature Request] Make editor canvas take as much screen space as possible

See discussion https://github.com/huchenlei/sd-webui-openpose-editor/discussions/20#discussioncomment-6151840
This feature should make the fabric canvas behaves like a photoshop window. The window size is fixed, while the canvas inside the window can be zoomed in/out, and change H/W ratio, change size, etc.

Implemented in v0.1.5.

repo: huchenlei/sd-webui-openpose-editor


text:
Data thats downloaded seems to be limited

For example, for the general election polls, downloading all the data only yields five or six polls where Trump and Clinton are > 0. Is there a reason for this?

Hi there! Please provide some example code, the result python-pollster gives, and a description of what's wrong with that result.

So, I looked at the chart:

And tried to download the corresponding polls:

But, the number of non-trivial results seem to be very limited:

In fact, for significant polls there are barely 60:

Just wanted to make sure that made sense

Leads to sparse maps:
https://github.com/bloomberg/bqplot/pull/303

Thanks ... but what's the code you wrote to populate data?

For getting the data I used:
raw_data = pollster.charts(topic='2016-president')

def get_candidate_data(question):
    clinton, trump, undecided, other = 0., 0., 0., 0.
    for candidate in question['subpopulations'][0]['responses']:
        if candidate['choice'] == 'Hillary Clinton':
            clinton = candidate['value']
        elif candidate['choice'] == 'Donald Trump':
            trump = candidate['value']
        elif candidate['choice'] == 'Undecided':
            undecided = candidate['value']
        else:
            other = candidate['value']
    return clinton, trump, other, undecided

def get_row(question, partisan='Nonpartisan', end_date='2016-06-21'):
    if question['topic'] != '2016-president':
        return
    hillary, donald, other, undecided = get_candidate_data(question)
    return [{'Name': question['name'], 'Partisan': partisan, 'State': question['state'],
             'Date': np.datetime64(end_date), 'Trump': donald, 'Clinton': hillary, 'Other': other,
             'Undecided': undecided}]

def analyze_polls(polls):
    global data
    for poll in polls:
        for question in poll.questions:
            resp = get_row(question, partisan=poll.partisan, end_date=poll.end_date)
            if resp is not None:
                data = data.append(resp)
    return

data = pd.DataFrame(columns=['Name', 'Partisan', 'State', 'Date', 'Trump', 'Clinton', 'Other',
                     'Undecided'])

for i in raw_data:
    analyze_polls(i.polls())

You're going poll-question by poll-question, but different poll questions have different response phrasings. For some polls, a choice might be u'Hillary Clinton'. For other polls, the choice could be u'Clinton'. Another would be u'Hillary Clinton, the Democratic Candidate'. Pollster doesn't control these: it just copy/pastes from the poll results. If there's a typo in the poll, there will be a typo in Pollster.
So here's a simple workaround: look for last_name of u'Trump' or u'Clinton'. That will work because last_name is a field that we at Pollster control.
We're totally on board with helping you use the API. But I don't consider this a bug in python-pollster, so I'll close this issue. I encourage you to join https://groups.google.com/forum/#!forum/pollster-users and discuss everything API-related.

That worked much better @adamhooper
Thanks! Just a quick question, whats the license on the data?

CC-by-nc-sa , as per http://elections.huffingtonpost.com/pollster/api.
IANAL, but my understanding is: "by" is easy (please cite and link to us) and "sa" is easy (please publish derived CSVs -- which is good form, anyway). As for NC: I believe AOL considers public-interest news reporting (like NYT or 538 predictions/stories) to be "non-commercial". I don't know where the boundaries are to that definition. To be safe, please double-check with pollster-feedback@huffingtonpost.com.

authors: adamhooper dmadeka

repo: huffpostdata/python-pollster


text:
🛑 VGMdb API is down

In 690854f, VGMdb API (https://vgmdb.info) was down:

HTTP code: 502
Response time: 333 ms

Resolved: VGMdb API is back up in fd24595 after 8 minutes.

repo: hufman/upptime


text:
🛑 Akkoma is down

In 98de4d6, Akkoma (https://akkoma.chez.chat) was down:

HTTP code: 404
Response time: 734 ms

Resolved: Akkoma is back up in 051f8a4 after 9 minutes.


text:
strict_slashes is not respected from the application level

Describe the bug
This issue is porting from huge-success/sanic-openapi#102. When using Sanic(strict_slashes=True), it won't be apply to any blueprint.
Code snippet
from sanic import Sanic, Blueprint
from sanic.response import text

app = Sanic(strict_slashes=True)

bp = Blueprint("bp")

@bp.get("/test")
async def test(request):
    return text("test")

app.blueprint(bp)

for route in app.router.routes_all:
    print(route)

Output:
/test
/test/

Expected behavior
The strict_slashes in app level should be respected until user override it by setting Blueprint(..., strict_slashes=True) or in blueprint's route level.
Environment (please complete the following information):

OS: Ubuntu 18.04
Version: master(https://github.com/huge-success/sanic/commit/e36f398aa689cb1852a171f9e6e60f78d8fa21d5)

Additional context
None

I'm working on a new router idea that would, among other things, address this. So unless anyone else wants to address this separately, I'll have a fix for this in my proposed router.

Cool, expecting your awesome work.

@chenjr0719.... the pressure is on 😱

@ahopkins Oops, Please don't feel any pressure. If there are any help I can provide, please let me know. :wink:

@chenjr0719 ;-) Was just joking. When I am done with the POC for it (probably not till next week at the earliest), I will definitely be looking for as much feedback and thoughts as possible since it will potentially be a big change (I am trying to make sure there will not be any API changes though).

@chenjr0719  @ahopkins  I think we might have to think a bit more on this specific issue. I managed to fix this with minor code changes. But this can be a breaking change in some cases.
app = Sanic()

@app.get("/test", strict_slashes=True)
def test(request):
    return text("OK")

assert app.test_client.get("/test")[1].status == 200

This was a valid behavior previously. But considering that there is a strict_slashes attribute, the behavior seems odd and needs to be fixed. Only /test/ should be a valid URL that will return 200 and /test should return 404 instead.
Also, what order do we take into account while overriding the strict_slashes behavior? i.e. Which takes priority?
app = Sanic(strict_slashes=True)

@app.get("/test")
def test(request):
    return  text("Test")

bp = Blueprint("bp")

@bp.get("/one", strict_slashes=False)
def one(request):
    return text("one")

@bp.get("/second")
def second(request):
    return text("second")

app.blueprint(bp)

print(app.router.route_all)

bp2 = Blueprint("bp2", strict_slashes=False)

@bp2.get("/third")
def third(request):
    return text("third")

What is the expected behavior in the above case?

@harshanarayana I agree, which is sort of why I am not sure we tackle it separate from any other router changes we want to make. I think it would be best to make them all at once.

@ahopkins Works for me. However, I just opened a new PR with WIP tag to address this + the other dependency issue. Take a look when you can and let me know your feedback. It's pending few documentation changes
@chenjr0719

@harshanarayana
What I think is like this hierarchy structure.
Application level - True/False(default)
├── Route level - None(default)/True/False
└── Blueprint level - None(default)/True/False
        └── BP's route level - None(default)/True/False

So, in the Python world, it would be:
# route
if route.strict_slashes is None:
    route.strict_slashes = app.strict_slashes

# blueprint route
if bp_route.strict_slashes is None:
    if bp.strict_slashes is None:
        bp_route.strict_slashes = app.strict_slashes
    else:
        bp_route.strict_slashes = bp.strict_slashes

Now, back to your cases:
from sanic import Sanic, Blueprint
from sanic.response import text

app = Sanic(strict_slashes=True)

@app.get("/test")
def test(request):
    return text("Test")

# this should follow app level
assert app.test_client.get("/test")[1].status == 200
assert app.test_client.get("/test/")[1].status == 404

bp = Blueprint("bp")

@bp.get("/one", strict_slashes=False)
def one(request):
    return text("one")

# this should follow bp route level
assert app.test_client.get("/one")[1].status == 200
assert app.test_client.get("/one/")[1].status == 200

@bp.get("/second")
def second(request):
    return text("second")

# this should follow bp level, but the bp level is None, so follow app level
assert app.test_client.get("/second")[1].status == 200
assert app.test_client.get("/second/")[1].status == 404

app.blueprint(bp)

bp2 = Blueprint("bp2", strict_slashes=False)

@bp2.get("/third")
def third(request):
    return text("third")

# this should follow bp level
assert app.test_client.get("/third")[1].status == 200
assert app.test_client.get("/third/")[1].status == 404

What do you think?

@chenjr0719 Let me get back to you on this by EOD today. I am caught in the middle of a training session and haven't been able to look at it.

@chenjr0719 That looks awesome. I agree with the results of each. Without having the test cases in front of me, there are a few permutations that are still not accounted for in your layout. We should make sure to test them all for consistency.

Application level - True/False(default)
├── Route level - None(default)/True/False
└── Blueprint level - None(default)/True/False
└── BP's route level - None(default)/True/False

In most of the places, the default argument is None but we override them to False as the first action inside the method. Provided I change that, the precedence of check can be as follows
route level
|___ Blueprint level
       |__ App Level

When deciding the behavior for the strict_slashes, we traverse the above order and if we find a non None value at any level, then we take that as the actual value for that specific route.
def _check_if_strict_slash_should_be_enforced(slash_options):
    for op_type in ["route", "bp", "app"]:
        if slash_options.get(op_type) is not None:
            return slash_options.get(op_type)
    else:
        return False

Case 1
app = Sanic(strict_slashes=True)

@app.get("/test")
def test(request):
    return text("Test")

assert app.test_client.get("/test")[1].status == 200  # I think this behavior should be 404
assert app.test_client.get("/test/")[1].status == 404 # I think this behavior should be 200

In above, I think the /test endpoint needs to get a 404 and /test/ needs to get a 200. Since the strict_slashes is enforced at the app level, even if the route registration is missing the trailing /, it can be added while route registering. Otherwise, the app level strict_slashes will not have any value.
Case 2
app = Sanic(strict_slashes=True)
bp = Blueprint("bp")

@bp.get("/one", strict_slashes=False)
def one(request):
    return text("one")

# this should follow bp route level
assert app.test_client.get("/one")[1].status == 200
assert app.test_client.get("/one/")[1].status == 200

Agree on this. The strict_slashes at the BP Route level takes over and enforces the non strict_slashes behavior.
Case 3
app = Sanic(strict_slashes=True)

bp2 = Blueprint("bp2", strict_slashes=False)

@bp2.get("/third")
def third(request):
    return text("third")

app.blueprint(bp2)

# this should follow bp level
assert app.test_client.get("/third")[1].status == 200
assert app.test_client.get("/third/")[1].status == 404 # I think this should return 200 as well.

In above case the Blueprint level's strict_slashes should come into play causing the registration to follow the non strict_slashes behavior and both /third and /third/ should be valid.
General Items

If there is a strict_slashes=True anywhere in the hierarchy that we ultimately decide upon, every URL needs to terminate with a trailing / irrespective of how they are represented in the Route level.
If you want to override the strict_slashes behavior you can individually do that either at the route or BP level while creating the routes
strict_slashes once applied at the BP or Application level are set for the lifetime of the object. So technically, DO NOT change this value once decided. You can override them individually at route level as required.

@harshanarayana

If you want to override the strict_slashes behavior you can individually do that either at the route or BP level while creating the routes

strict_slashes once applied at the BP or Application level are set for the lifetime of the object. So technically, DO NOT change this value once decided. You can override them individually at route level as required.

Totally agree both.

If there is a strict_slashes=True anywhere in the hierarchy that we ultimately decide upon, every URL needs to terminate with a trailing / irrespective of how they are represented in the Route level.

I don't think this is strict_slashes supposed to do. At current implementation, It depends on the uri of the route.
When the strict_slashes=False, no matter your uri ends with slash or not, it will create two endpoints, one with slash, another one without slash.
app = Sanic(strict_slashes=False)

@app.get("/test")
def test(request):
    return text("Test")

assert app.test_client.get("/test")[1].status == 200
assert app.test_client.get("/test/")[1].status == 200

When the strict_slashes=True and the uri ends without slash, there will be only one endpoint /test be created.
app = Sanic(strict_slashes=True)

@app.get("/test")
def test(request):
    return text("Test")

assert app.test_client.get("/test")[1].status == 200
assert app.test_client.get("/test/")[1].status == 404

In the other hand, when the uri ends with slash, it will be opposite to the above case.
app = Sanic(strict_slashes=True)

@app.get("/test/")
def test(request):
    return text("Test")

assert app.test_client.get("/test")[1].status == 404
assert app.test_client.get("/test/")[1].status == 200

I like your idea about when strict_slashes=True all uri should ends with slash. It can simplify most cases. But, it also changes the behavior of strict_slashes totally. I'm afraid it will cause a big impact to users.
I suggest to separate current discussion to two parts, one is about the override mechanism, another is how strict_slashes works.

About the override mechanism, I think we both agree it should be route -> blueprint -> application.
About how strict_slashes works, I have to say that we need more time and feedback to determine should we change it's behavior or not. And I think this might also related to #1420.

Yes, it would be a change in behavior. Which leads to hesitancy and reluctance to make a change. However, it also seems a bit counter intuitive.Besides the consistency aspect, if there is to be a change in behavior, we should do a survey of other frameworks. Being intuitive is also important.

@chenjr0719 @ahopkins  Agreed. I am changing the original PR to only include the changes required for addressing the strict_slashes option inheritance against the defined hierarchy.
As for the change of default behavior implemented currently, let me move that discussion to the thread in community and see what comes out of it.

Closing this. Any further discussion, let's move to forums.

authors: ahopkins chenjr0719 harshanarayana

repo: huge-success/sanic


text:
pip3 install fails on version 19.6.2 on Mac

When trying to install the package globally:
pip3 install sanic
It says in the end:
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/LICENSE.md'
Consider using the `--user` option or check the permissions.

It is expected that access to this directory requires root permissions but it is not clear why setup process tries to write there.
Same works if I specify the previous version 19.6.0.

OS: macOS 10.14.5
Python3 installed through brew

Hi!
You're trying to install Sanic using your user's "global" install of Python that you installed with brew. This won't work because, like the error message says, it tries to install into an area that requires elevated permissions.
This is best fixed one of two ways:

Using a virtual environment with a tool like venv, virtualenv, or pipenv
OR using the --user flag, which will install it locally for your user.

If you're new to Python, you should definitely consider using a virtual environment. It keeps your dependencies isolated so you can work on multiple projects without conflict!
Here is a link to using venv in the python docs.

MacOS brew normally installs in /usr/local/ where the normal user has write permissions. Perhaps pip3 install sanic should be installing to /usr/local/ rather than /usr/?
@greatvovan Which python3/pip3 are being used on your system? Use which pip3 or which python3 to find out. Are these in /usr/local/bin or /usr/bin?

While this is dupable, there are other things that fail using pip/pip3 due to permissions, and not just on MacOS. I wouldn't really call this a sanic issue, and I would endorse using the --user flag.
Alternately, you can force install it globally with sudo pip3 install sanic - but I really wouldn't recommend that.

authors: Tronic greatvovan madelyneriksen sjsadowski


text:
The multi process lr scheduler implementation doesn't seem right

https://github.com/huggingface/accelerate/blob/074d8d5a5a131501cc0a2a690cc14337395dcb09/src/accelerate/scheduler.py#L72-L75
The scheduler should not step if self.scheduler._step_count > self.scheduler.total_steps.

Thanks for the notice! Will be fixed by #849

authors: ernestchu muellerzr

repo: huggingface/accelerate


text:
Error: Metal error no metal implementation for softmax-last-dim

Cargo run error while running Phi-2 model in Macbook Pro M2.
cargo run

Error ->
Error: Metal error no metal implementation for softmax-last-dim
Caused by:
    no metal implementation for softmax-last-dim

.toml file:
# other dep
accelerate-src = { version = "0.3.2" }
candle-core = { git = "https://github.com/huggingface/candle.git", version = "0.3.0", features = ["metal"] }

There is no implementation for this, or I should use a specific branch for metal?

And when using accelerate, I'm getting this:
Running on CPU, to run on GPU(metal), build this example with `--features metal`

I'm having a similar issue. M2 Macbook pro. The following metal-kernels tests are failing.

Regarding the tests, could it be because the bfloat16 type was added to Metal in version 14 of macOS? Can you run sw_vers in your terminal?

@pcuenca you are right, thank you

I had a similar issue, and the solution was quite simple - enabled metal feature for all dependencies
"candle-core/metal", "candle-nn/metal", "candle-transformers/metal"

Example:
candle-core = { git = "https://github.com/huggingface/candle.git", version = "0.3.3", features = ["metal"]}
candle-transformers = { git = "https://github.com/huggingface/candle.git", version = "0.3.3" , features = ["metal"]}
candle-nn = { git = "https://github.com/huggingface/candle.git", version = "0.3.3" , features = ["metal"]}

Thanks :)
Maybe off-topic question: how about mkl on intel graphic card? (Mac of course)

repo: huggingface/candle


text:
Performance Issue while using Metal backend.

Hi there,
Candle is really a great project when using Rust. But currently I faced some performance Issue when I use it to infer my Model.
Here is some of my model code:
impl Module for RRDB {
    fn forward(&self, x: &Tensor) -> Result<Tensor> {
        let start = Instant::now();
        let oup = self.rdb1.forward(x)?;
        let oup = self.rdb2.forward(&oup)?;
        let oup = self.rdb3.forward(&oup)?;
        let res = oup * 0.2 + x;
        println!("block process time: {:?}", start.elapsed());
        res
    }
}

After I do this forward function manny times, the time consuming is increasing. Here is the log:
block process time: 2.047125ms
block process time: 1.727084ms
block process time: 1.85325ms
block process time: 1.602167ms
block process time: 1.691666ms
block process time: 1.804833ms
block process time: 1.78675ms
block process time: 1.742667ms
block process time: 1.581042ms
block process time: 1.644833ms
block process time: 1.772917ms
block process time: 1.695958ms
block process time: 1.766334ms
block process time: 1.623375ms
block process time: 1.742208ms
block process time: 1.675958ms
block process time: 1.771334ms
block process time: 236.509ms
block process time: 503.890583ms
block process time: 390.076375ms
block process time: 495.903084ms
block process time: 529.992666ms
block process time: 366.003875ms
block process time: 495.694083ms
block process time: 390.461375ms
block process time: 497.969958ms
block process time: 498.7325ms
block process time: 403.164125ms
block process time: 488.229458ms
block process time: 524.081875ms
block process time: 375.679167ms
block process time: 508.565042ms
block process time: 389.132209ms
block process time: 508.387208ms
block process time: 500.397959ms
block process time: 401.458583ms
block process time: 490.374167ms
block process time: 518.663584ms
block process time: 376.272792ms
block process time: 518.60275ms
block process time: 1.472589375s
block process time: 328.865375ms
block process time: 322.741459ms
block process time: 250.503333ms
block process time: 316.479292ms
block process time: 317.622416ms
block process time: 242.031125ms
block process time: 326.601459ms
block process time: 251.282875ms
block process time: 323.213208ms
block process time: 319.51025ms
block process time: 256.726084ms
block process time: 314.952541ms
block process time: 330.990334ms
block process time: 240.115625ms
block process time: 325.475333ms
block process time: 249.119167ms
block process time: 326.153334ms
block process time: 319.268833ms
block process time: 255.009291ms
block process time: 314.721042ms
block process time: 333.955625ms
block process time: 241.245666ms
block process time: 1.16066575s
block process time: 443.387583ms
block process time: 566.287125ms
block process time: 554.786625ms
block process time: 431.030416ms
block process time: 544.772083ms
block process time: 568.671916ms
block process time: 410.647042ms
block process time: 558.133167ms
block process time: 425.856291ms
block process time: 556.451292ms
block process time: 544.14825ms
block process time: 440.875167ms
block process time: 539.090792ms
block process time: 575.805542ms
block process time: 412.616417ms
block process time: 557.536041ms
block process time: 426.443416ms
block process time: 566.2835ms
block process time: 544.170708ms
block process time: 434.747875ms
block process time: 535.491166ms
block process time: 548.576334ms
block process time: 1.64796325s
block process time: 348.008166ms
block process time: 282.976ms
block process time: 348.38475ms
block process time: 366.042791ms
block process time: 267.093916ms
block process time: 346.409167ms
block process time: 348.611666ms
block process time: 284.104292ms
block process time: 360.060416ms
block process time: 271.554ms
block process time: 357.186167ms
block process time: 350.56875ms
block process time: 277.07075ms
block process time: 350.898ms
block process time: 350.061208ms
block process time: 282.163458ms
block process time: 346.33075ms
block process time: 368.18625ms
block process time: 263.653083ms
block process time: 358.285875ms
block process time: 273.358ms
block process time: 362.535459ms
block process time: 1.015997542s
block process time: 88.540916ms
block process time: 109.382833ms
block process time: 109.310416ms
block process time: 87.187625ms
block process time: 114.729167ms
block process time: 87.404291ms
block process time: 111.601917ms
block process time: 110.608084ms
block process time: 89.529708ms
block process time: 108.535625ms
block process time: 114.542375ms
block process time: 84.9315ms
block process time: 112.986166ms
block process time: 87.082083ms
block process time: 110.971542ms
block process time: 112.866ms
block process time: 89.563458ms
block process time: 108.4445ms
block process time: 115.158416ms
block process time: 83.165875ms
block process time: 113.004167ms
block process time: 87.176333ms
block process time: 344.457458ms
block process time: 72.254792ms
block process time: 56.214959ms
block process time: 71.726666ms
block process time: 70.487208ms
block process time: 57.035167ms

Does anyone know why the time consuming can change from 1ms to 500ms?

Just to mention something that may be a bit obvious: metal is an asynchronous api (same as cuda) so in order to time things properly you have to insert some "synchronise" calls to ensure that the code has fully run up to the point where you make the time measurements.

authors: Jasonsey LaurentMazare


text:
datasets cannot be installed alongside apache-beam

Describe the bug
If one installs apache-beam alongside datasets (which is required for the wikipedia dataset) in certain environments (such as a Google Colab notebook), they appear to install successfully, however, actually trying to do something such as importing the load_dataset method from datasets results in a crashing error.
I think the problem is that apache-beam version 2.49.0 requires dill>=0.3.1.1,<0.3.2, but the latest version of multiprocess (0.70.15) (on which datasets depends) requires  dill>=0.3.7,, so this is causing the dependency resolver to use an older version of multiprocess which leads to the datasets crashing since it doesn't actually appear to be compatible with older versions.
Steps to reproduce the bug
See this Google Colab notebook to easily reproduce the bug.
In some environments, I have been able to reproduce the bug by running the following in Bash:
$ pip install datasets apache-beam

then the following in a Python shell:
from datasets import load_dataset

Here is my stacktrace from running on Google Colab:

stacktrace
[/usr/local/lib/python3.10/dist-packages/datasets/__init__.py](https://localhost:8080/#) in <module>
     20 __version__ = "2.14.4"
     21
---> 22 from .arrow_dataset import Dataset
     23 from .arrow_reader import ReadInstruction
     24 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in <module>
     64
     65 from . import config
---> 66 from .arrow_reader import ArrowReader
     67 from .arrow_writer import ArrowWriter, OptimizedTypedSequence
     68 from .data_files import sanitize_patterns

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py](https://localhost:8080/#) in <module>
     28 import pyarrow.parquet as pq
     29
---> 30 from .download.download_config import DownloadConfig
     31 from .naming import _split_re, filenames_for_dataset_split
     32 from .table import InMemoryTable, MemoryMappedTable, Table, concat_tables

[/usr/local/lib/python3.10/dist-packages/datasets/download/__init__.py](https://localhost:8080/#) in <module>
      7
      8 from .download_config import DownloadConfig
----> 9 from .download_manager import DownloadManager, DownloadMode
     10 from .streaming_download_manager import StreamingDownloadManager

[/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py](https://localhost:8080/#) in <module>
     33 from ..utils.info_utils import get_size_checksum_dict
     34 from ..utils.logging import get_logger, is_progress_bar_enabled, tqdm
---> 35 from ..utils.py_utils import NestedDataStructure, map_nested, size_str
     36 from .download_config import DownloadConfig
     37

[/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <module>
     38 import dill
     39 import multiprocess
---> 40 import multiprocess.pool
     41 import numpy as np
     42 from packaging import version

[/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py](https://localhost:8080/#) in <module>
    607 #
    608
--> 609 class ThreadPool(Pool):
    610
    611     from .dummy import Process

[/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py](https://localhost:8080/#) in ThreadPool()
    609 class ThreadPool(Pool):
    610
--> 611     from .dummy import Process
    612
    613     def __init__(self, processes=None, initializer=None, initargs=()):

[/usr/local/lib/python3.10/dist-packages/multiprocess/dummy/__init__.py](https://localhost:8080/#) in <module>
     85 #
     86
---> 87 class Condition(threading._Condition):
     88     # XXX
     89     if sys.version_info < (3, 0):

AttributeError: module 'threading' has no attribute '_Condition'

I've also found that attempting to install these datasets and apache-beam in certain environments (e.g. via pip inside a conda env) simply causes pip to hang indefinitely.
Expected behavior
I would expect to be able to import methods from datasets without crashing. I have tested that this is possible as long as I do not attempt to install apache-beam.
Environment info
Google Colab

I noticed that this is actually covered by #5613, which for some reason I didn't see when I searched the issues in this repo the first time.

repo: huggingface/datasets


text:
attribute error when writing trainer.train()

Describe the bug
AttributeError                            Traceback (most recent call last)
Cell In[39], line 2
1 # Start the training process
----> 2 trainer.train()
File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:1539, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
1537         hf_hub_utils.enable_progress_bars()
1538 else:
-> 1539     return inner_training_loop(
1540         args=args,
1541         resume_from_checkpoint=resume_from_checkpoint,
1542         trial=trial,
1543         ignore_keys_for_eval=ignore_keys_for_eval,
1544     )
File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:1836, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
1833     rng_to_sync = True
1835 step = -1
-> 1836 for step, inputs in enumerate(epoch_iterator):
1837     total_batched_samples += 1
1839     if self.args.include_num_input_tokens_seen:
File /opt/conda/lib/python3.10/site-packages/accelerate/data_loader.py:451, in DataLoaderShard.iter(self)
449 # We iterate one batch ahead to check when we are at the end
450 try:
--> 451     current_batch = next(dataloader_iter)
452 except StopIteration:
453     yield
File /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:630, in _BaseDataLoaderIter.next(self)
627 if self._sampler_iter is None:
628     # TODO(https://github.com/pytorch/pytorch/issues/76750)%3C/span%3E)
629     self._reset()  # type: ignore[call-arg]
--> 630 data = self._next_data()
631 self._num_yielded += 1
632 if self._dataset_kind == _DatasetKind.Iterable and
633         self._IterableDataset_len_called is not None and
634         self._num_yielded > self._IterableDataset_len_called:
File /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:674, in _SingleProcessDataLoaderIter._next_data(self)
672 def _next_data(self):
673     index = self._next_index()  # may raise StopIteration
--> 674     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
675     if self._pin_memory:
676         data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)
File /opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py:51, in _MapDatasetFetcher.fetch(self, possibly_batched_index)
49         data = self.dataset.getitems(possibly_batched_index)
50     else:
---> 51         data = [self.dataset[idx] for idx in possibly_batched_index]
52 else:
53     data = self.dataset[possibly_batched_index]
File /opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py:51, in (.0)
49         data = self.dataset.getitems(possibly_batched_index)
50     else:
---> 51         data = [self.dataset[idx] for idx in possibly_batched_index]
52 else:
53     data = self.dataset[possibly_batched_index]
File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:1764, in Dataset.getitem(self, key)
1762 def getitem(self, key):  # noqa: F811
1763     """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
-> 1764     return self._getitem(
1765         key,
1766     )
File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:1749, in Dataset._getitem(self, key, decoded, **kwargs)
1747 formatter = get_formatter(format_type, features=self.features, decoded=decoded, **format_kwargs)
1748 pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
-> 1749 formatted_output = format_table(
1750     pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
1751 )
1752 return formatted_output
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:540, in format_table(table, key, formatter, format_columns, output_all_columns)
538 else:
539     pa_table_to_format = pa_table.drop(col for col in pa_table.column_names if col not in format_columns)
--> 540     formatted_output = formatter(pa_table_to_format, query_type=query_type)
541     if output_all_columns:
542         if isinstance(formatted_output, MutableMapping):
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:281, in Formatter.call(self, pa_table, query_type)
279 def call(self, pa_table: pa.Table, query_type: str) -> Union[RowFormat, ColumnFormat, BatchFormat]:
280     if query_type == "row":
--> 281         return self.format_row(pa_table)
282     elif query_type == "column":
283         return self.format_column(pa_table)
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/torch_formatter.py:57, in TorchFormatter.format_row(self, pa_table)
56 def format_row(self, pa_table: pa.Table) -> dict:
---> 57     row = self.numpy_arrow_extractor().extract_row(pa_table)
58     return self.recursive_tensorize(row)
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:154, in NumpyArrowExtractor.extract_row(self, pa_table)
153 def extract_row(self, pa_table: pa.Table) -> dict:
--> 154     return _unnest(self.extract_batch(pa_table))
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:160, in NumpyArrowExtractor.extract_batch(self, pa_table)
159 def extract_batch(self, pa_table: pa.Table) -> dict:
--> 160     return {col: self._arrow_array_to_numpy(pa_table[col]) for col in pa_table.column_names}
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:160, in (.0)
159 def extract_batch(self, pa_table: pa.Table) -> dict:
--> 160     return {col: self._arrow_array_to_numpy(pa_table[col]) for col in pa_table.column_names}
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:196, in NumpyArrowExtractor._arrow_array_to_numpy(self, pa_array)
194         array: List = pa_array.to_numpy(zero_copy_only=zero_copy_only).tolist()
195 if len(array) > 0:
--> 196     if any(
197         (isinstance(x, np.ndarray) and (x.dtype == np.object or x.shape != array[0].shape))
198         or (isinstance(x, float) and np.isnan(x))
199         for x in array
200     ):
201         return np.array(array, copy=False, **{**self.np_array_kwargs, "dtype": np.object})
202 return np.array(array, copy=False, **self.np_array_kwargs)
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:197, in (.0)
194         array: List = pa_array.to_numpy(zero_copy_only=zero_copy_only).tolist()
195 if len(array) > 0:
196     if any(
--> 197         (isinstance(x, np.ndarray) and (x.dtype == np.object or x.shape != array[0].shape))
198         or (isinstance(x, float) and np.isnan(x))
199         for x in array
200     ):
201         return np.array(array, copy=False, **{**self.np_array_kwargs, "dtype": np.object})
202 return np.array(array, copy=False, **self.np_array_kwargs)
File /opt/conda/lib/python3.10/site-packages/numpy/init.py:324, in getattr(attr)
319     warnings.warn(
320         f"In the future np.{attr} will be defined as the "
321         "corresponding NumPy scalar.", FutureWarning, stacklevel=2)
323 if attr in former_attrs:
--> 324     raise AttributeError(former_attrs[attr])
326 if attr == 'testing':
327     import numpy.testing as testing
AttributeError: module 'numpy' has no attribute 'object'.
np.object was a deprecated alias for the builtin object. To avoid this error in existing code, use object by itself. Doing this will not modify any behavior and is safe.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecationsAttributeError                            Traceback (most recent call last)
Cell In[39], line 2
1 # Start the training process
----> 2 trainer.train()
File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:1539, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
1537         hf_hub_utils.enable_progress_bars()
1538 else:
-> 1539     return inner_training_loop(
1540         args=args,
1541         resume_from_checkpoint=resume_from_checkpoint,
1542         trial=trial,
1543         ignore_keys_for_eval=ignore_keys_for_eval,
1544     )
File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:1836, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
1833     rng_to_sync = True
1835 step = -1
-> 1836 for step, inputs in enumerate(epoch_iterator):
1837     total_batched_samples += 1
1839     if self.args.include_num_input_tokens_seen:
File /opt/conda/lib/python3.10/site-packages/accelerate/data_loader.py:451, in DataLoaderShard.iter(self)
449 # We iterate one batch ahead to check when we are at the end
450 try:
--> 451     current_batch = next(dataloader_iter)
452 except StopIteration:
453     yield
File /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:630, in _BaseDataLoaderIter.next(self)
627 if self._sampler_iter is None:
628     # TODO(https://github.com/pytorch/pytorch/issues/76750)%3C/span%3E)
629     self._reset()  # type: ignore[call-arg]
--> 630 data = self._next_data()
631 self._num_yielded += 1
632 if self._dataset_kind == _DatasetKind.Iterable and
633         self._IterableDataset_len_called is not None and
634         self._num_yielded > self._IterableDataset_len_called:
File /opt/conda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:674, in _SingleProcessDataLoaderIter._next_data(self)
672 def _next_data(self):
673     index = self._next_index()  # may raise StopIteration
--> 674     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
675     if self._pin_memory:
676         data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)
File /opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py:51, in _MapDatasetFetcher.fetch(self, possibly_batched_index)
49         data = self.dataset.getitems(possibly_batched_index)
50     else:
---> 51         data = [self.dataset[idx] for idx in possibly_batched_index]
52 else:
53     data = self.dataset[possibly_batched_index]
File /opt/conda/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py:51, in (.0)
49         data = self.dataset.getitems(possibly_batched_index)
50     else:
---> 51         data = [self.dataset[idx] for idx in possibly_batched_index]
52 else:
53     data = self.dataset[possibly_batched_index]
File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:1764, in Dataset.getitem(self, key)
1762 def getitem(self, key):  # noqa: F811
1763     """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
-> 1764     return self._getitem(
1765         key,
1766     )
File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:1749, in Dataset._getitem(self, key, decoded, **kwargs)
1747 formatter = get_formatter(format_type, features=self.features, decoded=decoded, **format_kwargs)
1748 pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
-> 1749 formatted_output = format_table(
1750     pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
1751 )
1752 return formatted_output
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:540, in format_table(table, key, formatter, format_columns, output_all_columns)
538 else:
539     pa_table_to_format = pa_table.drop(col for col in pa_table.column_names if col not in format_columns)
--> 540     formatted_output = formatter(pa_table_to_format, query_type=query_type)
541     if output_all_columns:
542         if isinstance(formatted_output, MutableMapping):
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:281, in Formatter.call(self, pa_table, query_type)
279 def call(self, pa_table: pa.Table, query_type: str) -> Union[RowFormat, ColumnFormat, BatchFormat]:
280     if query_type == "row":
--> 281         return self.format_row(pa_table)
282     elif query_type == "column":
283         return self.format_column(pa_table)
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/torch_formatter.py:57, in TorchFormatter.format_row(self, pa_table)
56 def format_row(self, pa_table: pa.Table) -> dict:
---> 57     row = self.numpy_arrow_extractor().extract_row(pa_table)
58     return self.recursive_tensorize(row)
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:154, in NumpyArrowExtractor.extract_row(self, pa_table)
153 def extract_row(self, pa_table: pa.Table) -> dict:
--> 154     return _unnest(self.extract_batch(pa_table))
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:160, in NumpyArrowExtractor.extract_batch(self, pa_table)
159 def extract_batch(self, pa_table: pa.Table) -> dict:
--> 160     return {col: self._arrow_array_to_numpy(pa_table[col]) for col in pa_table.column_names}
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:160, in (.0)
159 def extract_batch(self, pa_table: pa.Table) -> dict:
--> 160     return {col: self._arrow_array_to_numpy(pa_table[col]) for col in pa_table.column_names}
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:196, in NumpyArrowExtractor._arrow_array_to_numpy(self, pa_array)
194         array: List = pa_array.to_numpy(zero_copy_only=zero_copy_only).tolist()
195 if len(array) > 0:
--> 196     if any(
197         (isinstance(x, np.ndarray) and (x.dtype == np.object or x.shape != array[0].shape))
198         or (isinstance(x, float) and np.isnan(x))
199         for x in array
200     ):
201         return np.array(array, copy=False, **{**self.np_array_kwargs, "dtype": np.object})
202 return np.array(array, copy=False, **self.np_array_kwargs)
File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:197, in (.0)
194         array: List = pa_array.to_numpy(zero_copy_only=zero_copy_only).tolist()
195 if len(array) > 0:
196     if any(
--> 197         (isinstance(x, np.ndarray) and (x.dtype == np.object or x.shape != array[0].shape))
198         or (isinstance(x, float) and np.isnan(x))
199         for x in array
200     ):
201         return np.array(array, copy=False, **{**self.np_array_kwargs, "dtype": np.object})
202 return np.array(array, copy=False, **self.np_array_kwargs)
File /opt/conda/lib/python3.10/site-packages/numpy/init.py:324, in getattr(attr)
319     warnings.warn(
320         f"In the future np.{attr} will be defined as the "
321         "corresponding NumPy scalar.", FutureWarning, stacklevel=2)
323 if attr in former_attrs:
--> 324     raise AttributeError(former_attrs[attr])
326 if attr == 'testing':
327     import numpy.testing as testing
AttributeError: module 'numpy' has no attribute 'object'.
np.object was a deprecated alias for the builtin object. To avoid this error in existing code, use object by itself. Doing this will not modify any behavior and is safe.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
Please help me to resolve the above error
Steps to reproduce the bug
Please resolve the issue of deprecated function np.object to object in the numpy
Expected behavior
np.object should be written as object only
Environment info
kaggle notebook

Hi! Kaggle notebooks use an outdated version of datasets, so you should update the datasets installation (with !pip install -U datasets) to avoid the error.

Thank you for your response
On Thu, Feb 29, 2024 at 10:55 PM Mario Šaško @.***>
wrote:

Closed #6669 https://github.com/huggingface/datasets/issues/6669 as
completed.
—
Reply to this email directly, view it on GitHub
https://github.com/huggingface/datasets/issues/6669#event-11969246964,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A2Y44YG2RRVMYONNKPLBVE3YV5SAPAVCNFSM6AAAAABDLZ3BTSVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRHE3DSMRUGY4TMNA
.
You are receiving this because you authored the thread.Message ID:
@.***>


text:
Descriptive error message on use_auth_token for gated models

Is your feature request related to a problem? Please describe.
When using from_pretrained or analogous for a gated model (like from_pretrained("CompVis/stable-diffusion-v1-4")) - if an authentication error is thrown, it is not descriptive to the user that they have to authenticate through the gate.

Describe the solution you'd like
Give a descriptive error to the user with a link to the gated model.
For restricted access there can be many failure cases:

Person did not use use_auth_token=True
Person did use use_auth_token=True but is logged out
Person did use use_auth_token=True, is logged in, but did not accept the gated model access
Person did use use_auth_token=YOUR_TOKEN with a valid token, but did not accept the gated model access

I feel like at least for the two last ones (but potentially all?), the link should be shown to the user to the org and tell people they have to read and accept the conditions in the gated model before accessing.

Another example for the issue: https://stackoverflow.com/questions/73451147/stablediffusion-colab-how-to-make-sure-youre-logged-in-with-huggingface-cli