text:
Port new features or bugfixes from Source

In the discussion below we can talk about commits to add to this to-port list.
Paste the whole commit text even if you intend to port only a part of its code.
When a commit is ported, reference to this issue in the commit message and add a comment to this issue specifying which functionality was ported from which commit.
This message will be edited and the text made strikedthrough.

To be tested more and eventually port.
Revision: a622737b2be9495d264091d7632af4987bfbfcfd
Author: Coruja coruja@outlook.com
Date: 13/06/2016 22:58:50
Message:
Fixed: Boat/ship parts resetting TYPE after use 'turn' commands.
To be tested more and eventually port.
Revision: acc1eb61d49e9c8d898027f64eafb8ee9076d343
Author: Coruja coruja@outlook.com
Date: 17/06/2016 09:11:42
Message:
Fixed: Paralyze / Paralyze Field making NPCs flood too many attack actions
Fixed: Console error when for some reason an NPC got created/moved to an invalid region
To be tested more and eventually port.
Revision: ffc3a52226fa3fa65ac40b3bf5476e5aa2e4f571
Author: Coruja coruja@outlook.com
Date: 22/06/2016 18:24:07
Message:
Added: Support for colored multis on target functions (only compatible with HS clients 7.0.9.0+)
Revision: aada1eb2fcb12b5b3564be6aa5a7b23e5d91a9e8
Author: Coruja coruja@outlook.com
Date: 27/06/2016 23:44:21
Message:
Fixed: Function f_onchar_delete not being called if the player char get deleted in any other way different from client Character Selecion menu
To be tested more and eventually ported.
Revision: a79d64ef2bee889a3a63ac0a7d96f5defc0e0459
Author: Coruja coruja@outlook.com
Date: 27/06/2016 23:57:49
Message:
Fixed: Multi dynamic regions getting replaced by script static regions on server resync
Revision: 3066341740896269766206db690f76a3785290b1
Author: Coruja coruja@outlook.com
Date: 30/06/2016 00:41:34
Message:
Fixed: Attack/Kill command on pets allowing select the pet itself as target, making it attack his owner
Revision: f26b39b9b40fba76660c36d57baa8387d3613272
Author: Coruja coruja@outlook.com
Date: 22/07/2016 22:52:03
Message:
Fixed: Security issue setting account login as chat name when newest clients try to setup the chat window for the first time and the char name is not available
Fixed: Invisible chars being incorrectly revealed by REVEALF_LOOTINGOTHERS reveal flag when picking items from the ground
Fixed: Return 0/1 on spell triggers not working correctly
Fixed: Function MOVENEAR not working correctly
To be discussed and eventually ported.
Revision: dd7a59b451e9d4d8d99c3bec15b330cf60dded2f
Author: Coruja coruja@outlook.com
Date: 02/08/2016 07:07:12
Message:
Fixed: Items inside trade window not firing @DropOn_Item trigger when the trade move the item to player bacpkack
Revision: 1fa0504b39e9df1074fb29121da2446926eb56b6
Author: Coruja coruja@outlook.com
Date: 03/08/2016 08:59:17
Message:
Changed: Max item capacity on containers changed from 255 to 125
This is required to make containers works properly on Enhanced Clients, because containers on these clients have hardcoded capacity of 125 items (OSI already uses this value since many years ago, even before enhanced client)
Revision: 5586c3db3ff6940a6d93c6d8062ccca1e195cbc2
Author: Coruja coruja@outlook.com
Date: 07/08/2016 06:54:07
Message:
Fixed: ARGN1 on char trigger @SkillPreStart not working correctly
Revision: 48ba5dc6e0e8be27b06208b31d128400eda47dd1
Author: Coruja coruja@outlook.com
Date: 12/08/2016 08:30:13
Message:
Fixed: Client war mode flag not being removed on death
To be tested more and eventually port.
Revision: 210563f9d22c4374e4faecdf92cd78ccd7114661
Author: Coruja coruja@outlook.com
Date: 13/08/2016 07:24:17
Message:
Fixed: Multi regions not reloading correctly after server resync
Also added a smart check to only reload it when needed (eg: AREADEF/ROOMDEF get changed on scripts)
To be tested more and eventually port.
Revision: c8223f203bbeb753ff0e1d3388db13e4df0b4d4a
Author: Coruja coruja@outlook.com
Date: 21/08/2016 05:29:13
Message:
Fixed: Internal check to prevent drop items inside walls preventing the item drop even when the wall is on another floor
Fixed: Message "Too many items here!" not showing correctly when items are dropped on areas with too many items
Revision: 30364c13a309452fe5fa2d28f264766c67b34e5d
Author: Coruja coruja@outlook.com
Date: 27/09/2016 02:07:55
Message:
Fixed: HTTP server not working correctly
Revision: 8fd75922705cbf38eda5130a65bbc91c020e418e
Author: ares ares@alathair.de
Date: 06/10/2016 22:31:42
Message:
As we have experienced at Alathair, exporting chars is nearly always meant to export other chars but not the own char, a flag here would be neccessary to control that. The default case should be not to export SRC itself. but it could be optionally turned on using the bitflag 0100. So if you want to export chars including yourself use flag 6. If it should include items use flag 7. Otherwise 1 2 and 3 will not include SRC.
...Continues...
To be tested more and eventually ported.
Revision: d66893bfe79e5e359f1d01465f24bdabbd515a6f
Author: ares ares@alathair.de
Date: 08/10/2016 20:10:14
Message:
Fixed a critical bug in background save mechanism where an unsaved offline player character disappears from worldsave when moved to an already saved sector.
Revision: 003321948c4694b5575c6072dec2d72db4a2e576
Author: Coruja coruja@outlook.com
Date: 17/10/2016 21:23:34
Message:
Fixed: NPCs losing 'statf_spawned' flag after server restart
To be tested more and eventually ported.
Revision: 72ff09966548b27829fcf2ed7608207ae6862d26
Author: Coruja coruja@outlook.com
Date: 12/11/2016 20:41:49
Message:
Fixed: Resurrect, Reveal, Meteor Swarm and Lightning spells showing effect animation even when EFFECT_ID=0 is set
[sphereCrypt.ini]: Added crypt key for classic clients 7.0.54 ~ 7.0.55 and enhanced clients 4.0.54 ~ 4.0.55
Revision: b054cd3277f26789ee2fbdd1cb7f96aa21d38127
Author: Coruja coruja@outlook.com
Date: 22/12/2016 19:09:09
Message:
Added: Missing buff icon for Criminal flag
I found this on ServUO repo, thanks for the help :P
Revision: bdc53e3cd70755a5add1c4b57819a2fef00d0245
Author: Coruja coruja@outlook.com
Date: 08/01/2017 19:23:28
Message:
Fixed: Function 'CRIMINAL 0' not updating char notoriety/buff when removing criminal flag
To be tested more and eventually ported.
Revision: 6e0f90f61fdd81d522143e35b6d968137c3d39ab
Author: Coruja coruja@outlook.com
Date: 25/01/2017 17:44:31
Message:
Fixed: Client encryption not being decrypted correctly on login process
Also reverted changes on packet 0BF 018 accidentally sent some commits ago
Revision: f1b5cffa0081d12aecf6600447ccae32cb752925
Author: Coruja coruja@outlook.com
Date: 21/12/2016 06:52:38
Message:
Fixed: Mass Curse spell not working correctly
Revision: 28b644628e76d3b837ecc960f5fb40d9bc920a40
Author: Coruja coruja@outlook.com
Date: 07/02/2017 21:59:56
Message:
Fixed: Char flag 'statf_hovering' (gargoyle fly ability) not clearing when gargoyle chars polymorph into non-gargoyle char ID

Probably more to add in the next days (i'm not done checking Source's changelog).

Closing, we are splitting ports into separate issues.

repo: Sphereserver/Source-experimental


text:
Devices 'Not Supported'

Hi,
Please tell me this project isn't dead. I use this every day and really love the work!
I've just had to rebuild the computer that runs zwave.me, home bridge and zway. I installed the latest version (3) and I'm getting some issues.
I added one device to test, a simple socket switch. Added fine on zway. Then installed the plugin on homebridge. Also when well. But on "home" the device is showing up 'Not supported' I think its not detected it currently. But it use to work fine before the rebuild.
Any help would be amazing.
Thanks,
Jamie

So, I had this exact problem, and was able to track it down. I don't know why it worked before for me as well.
For some reason, Zway appends the probe value to the deviceType; so switchBinary becomes switchBinary.switch, and so it doesn't find switchBinary in its big switch (no pun intended) statement that creates the details for the services. I noted it does the exact same thing for switchMultiLevel, but that works because it then maps switchMultilevel.multilevel back to switchMultiLevel, so I just added the switchBinary case to the same map, and it works fine.
I barely understand what I'm doing here, don't know if this is a correct fix for anyone else, and don't know how to deal with npm etc, so I'll leave it to someone else to decide if this is the right answer and submit a pull request (assuming anyone would respond to a PR).
specifically, add a line 89:     "switchBinary.switch": "switchBinary",
Here's a patch file with the same idea:
switchBinary.patch.txt
And here's my patched index.js, which on my Raspberry system goes in /usr/local/lib/node_modules/homebridge-zway
Homebridge-zway-index.js.zip
Again, your mileage may vary...

So, I had this exact problem, and was able to track it down. I don't know why it worked before for me as well.
For some reason, Zway appends the probe value to the deviceType; so switchBinary becomes switchBinary.switch, and so it doesn't find switchBinary in its big switch (no pun intended) statement that creates the details for the services. I noted it does the exact same thing for switchMultiLevel, but that works because it then maps switchMultilevel.multilevel back to switchMultiLevel, so I just added the switchBinary case to the same map, and it works fine.
I barely understand what I'm doing here, don't know if this is a correct fix for anyone else, and don't know how to deal with npm etc, so I'll leave it to someone else to decide if this is the right answer and submit a pull request (assuming anyone would respond to a PR).
specifically, add a line 89:  "switchBinary.switch": "switchBinary",
Here's a patch file with the same idea:
switchBinary.patch.txt
And here's my patched index.js, which on my Raspberry system goes in /usr/local/lib/node_modules/homebridge-zway
Homebridge-zway-index.js.zip
Again, your mileage may vary...

Great advice !!!
Thank you!

So, I had this exact problem, and was able to track it down. I don't know why it worked before for me as well.
For some reason, Zway appends the probe value to the deviceType; so switchBinary becomes switchBinary.switch, and so it doesn't find switchBinary in its big switch (no pun intended) statement that creates the details for the services. I noted it does the exact same thing for switchMultiLevel, but that works because it then maps switchMultilevel.multilevel back to switchMultiLevel, so I just added the switchBinary case to the same map, and it works fine.
I barely understand what I'm doing here, don't know if this is a correct fix for anyone else, and don't know how to deal with npm etc, so I'll leave it to someone else to decide if this is the right answer and submit a pull request (assuming anyone would respond to a PR).
specifically, add a line 89:  "switchBinary.switch": "switchBinary",
Here's a patch file with the same idea:
switchBinary.patch.txt
And here's my patched index.js, which on my Raspberry system goes in /usr/local/lib/node_modules/homebridge-zway
Homebridge-zway-index.js.zip
Again, your mileage may vary...

This works for my issue, too. Thanks.

I can't believe it took me a year to find this.  So incredibly frustrating....  That said... riddle me this.  I have a dozen BinarySwitches. All Jasco, some with the same software/firmware versions others a level up or down... I can discern no pattern as to why some show(ed) in Homekit correctly and others showed as Not supported.  Regardless making the change above resoled the issue for my remaining switches.

I just had to replace two Jasco switches that died, and replaced them with brand new Leviton Z-Wave switches. And, of course, after replacing them, the dreaded "Not Supported" started showing up again. I had added the previous fix to index.js (thanks, @KanG00!) but no dice. So I added some logging and figured out that these new Leviton switches were scanning as "switchBinary.power_switch_binary". That's just terrific. So, I added:
    "switchBinary.power_switch_binary": "switchBinary",
in the same place as the other "switchBinary.switch" addition, and it started working.
If I had more time, I'd figure out a way to make this not so brittle.

I just had to replace two Jasco switches that died, and replaced them with brand new Leviton Z-Wave switches (both model DZ15S). And, of course, after replacing them, the dreaded "Not Supported" started showing up again. I had added the previous fix to index.js (thanks, @KanG00!) but no dice. So I added some logging and figured out that these new Leviton switches were scanning as "switchBinary.power_switch_binary". That's just terrific. So, I added:
 "switchBinary.power_switch_binary": "switchBinary",
in the same place as the other "switchBinary.switch" addition, and it started working.
If I had more time, I'd figure out a way to make this not so brittle.

tl;dr
If you have this problem and don't want to change any code, use homebridge-zway-kevindayton specifically 0.6.0-alpha1 and tag your device with "Homebridge.Override.probeType:switch", this however makes custom tags not work.
If you want to run my code on HOOBS, install homebridge-zway-kevindayton then open your terminal and enter:
cd ~/.hoobs
npm install git+https://github.com/dkattan/homebridge-zway-kevindayton.git

Details
I just had to fix this after upgrading my Z-Way server from some ancient version to v3.2.2, and the fix that @mackworth posted did not work for my Jasco switches did not work, but the switch for the Leviton switches did work for my Jasco switches.
I'm running homebridge-zway-kevindayton 0.6.0-alpha1 which was released specifically to address this issue, but it no longer does due because the Z-Way API changed how it is returning device data.
Here's what I know, the Z-Way v3.2.2 API returns the following JSON for Jasco switches:
{
    "creationTime": 1643564474,
    "creatorId": 1,
    "customIcons": {},
    "deviceType": 'switchBinary',
    "firmware": '5.22',
    "h": -1928968201,
    "hasHistory": false,
    "id": 'ZWayVDev_zway_105-0-37',
    "location": 0,
    "locationName": 'globalRoom',
    "manufacturer": 'Jasco Products',
    "metrics": {
        "icon": 'switch',
        "isFailed": false,
        "title": 'Living Room Overhead Light',
        "level": 'off'
    },
    "nodeId": 105,
    "order": { rooms": 0, elements": 0, dashboard": 0
    },
    "permanently_hidden": false,
    "probeType": 'power_switch_binary',
    "product": '',
    "tags": [],
    "technology": 'Z-Wave',
    "visibility": true,
    "updateTime": 1643630461
}

In short, it appears that initially Z-Way did not return probeType for these switches. Then at some point in time it started returning it with the value "switch", which is why @mackworth 's fix worked, and why @kevindayton 's fix worked. (They are both doing the same thing in 2 different ways)
Then at some subsequent point later Z-Way began returning probeType: "power_switch_binary", and that is the state of affairs today in v3.2.2
If you peruse the release history, there is multiple mentions of probeType
https://z-wave.me/z-way/version-history/
Based on the date this issue was opened, I think the breaking changes occurred either in 18.07.2019 v3.0.0 or 03.04.2020 v3.0.5
Anyway, I incorporated changes for GE Gan Controllers as well as a fix for this issue in my branch, which I'll submit a PR for to homebridge-zway-kevindayton since this project appears to be abandoned.

source: gharchive/issue

authors: KanG00 dkattan iTommix jabeard3 jamoir kiwidyne mackworth

repo: SphtKr/homebridge-zway

license: isc

license_type: permissive

license_source: bigquery


text:
No suitable clads for Window Sensor

See the DEBUG log file attached. Obviously, there is no suitable class for the sensor:
logfile.txt

Hold on a bit, I'm going to try to push 0.5.0 out the door--and I think you may still be on 0.4.0 and not on the pre-release channel.

?????? the packet I installed is 0.5.0..... ????
Am 05.01.17 um 19:52 schrieb SphtKr:

Hold on a bit, I'm going to try to push 0.5.0 out the door--and I
think you may still be on 0.4.0 and not on the pre-release channel.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/SphtKr/homebridge-zway/issues/84#issuecomment-270725120,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOtuiBNCmZ6ieT0H1GGbDUhSGq_-OFI5ks5rPTv0gaJpZM4LZcuw.

--
Wolfgang Domröse
Von Eichendorff-Str. 23
37539 Bad Grund (Harz)

Oh...sorry, this is the Fibaro device and you're also on the conversation for #69. (I thought your log output looked like an old version but now I see it's not.)
Check the update over on the other issue...

authors: SphtKr WolfgangDomroese


text:
file uploads over 8k fail when using ModSecurity 2.7.5 and Nginx 1.4.2

Linux debian 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1+deb7u1 x86_64 GNU/Linux
ModSec 2.7.5 and Nginx 1.4.2
I have an Apache backend and it receives my file uploads and requests if the file is below 8k. Only got the basic modsecurity.conf loaded without any rules. If I set the SecRequestBodyAccess = Off even those pass through. Succesful upload:

[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Initialising transaction (txid AcAcAGl3AcAcAcAcAcAcAcAc).
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Transaction context created (dcfg 7f35a9f41980).
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Starting phase REQUEST_HEADERS.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Second phase starting (dcfg 7f35a9f41980).
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Input filter: Reading request body.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Multipart: Created temporary file 1 (mode 0600): /var/log/modsecurity_workdir/20130912-151049-AcAcAGl3AcAcAcAcAcAcAcAc-file-vIn5DC
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][5] Adding request argument (BODY): name "submit", value "Submit"
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Request body no files length: 150
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Input filter: Completed receiving request body (length 4719).
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Starting phase REQUEST_BODY.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Hook insert_filter: Adding input forwarding filter (r 7f35a9d950a0).
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Hook insert_filter: Adding output filter (r 7f35a9d950a0).
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Input filter: Forwarding input: mode=0, block=0, nbytes=-1 (f 7f35a9d962b0, r 7f35a9d950a0).
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Input filter: Forwarded 4719 bytes.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Input filter: Sent EOS.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Input filter: Input forwarding complete.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Starting phase RESPONSE_HEADERS.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Output filter: Not buffering response body for unconfigured MIME type "text/html".
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Output filter: Completed receiving response body (non-buffering).
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Starting phase RESPONSE_BODY.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Output filter: Output forwarding complete.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Initialising logging.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Starting phase LOGGING.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Recording persistent data took 0 microseconds.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Audit log: Ignoring a non-relevant request.
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Multipart: Cleanup started (remove files 1).
[12/Sep/2013:15:10:49 +0300] [/sid#7f35a9f410a0][rid#7f35a9d950a0][/upload_file.php][4] Multipart: Deleted file (part) "/var/log/modsecurity_workdir/20130912-151049-AcAcAGl3AcAcAcAcAcAcAcAc-file-vIn5DC"

failed upload:

[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Initialising transaction (txid AcAcATAcccAcAcRcvYAIpcAc).
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Transaction context created (dcfg 7f35a9f41980).
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Starting phase REQUEST_HEADERS.
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Second phase starting (dcfg 7f35a9f41980).
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Input filter: Reading request body.
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Multipart: Created temporary file 1 (mode 0600): /var/log/modsecurity_workdir/20130912-151248-AcAcATAcccAcAcRcvYAIpcAc-file-qmZcxo
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][5] Adding request argument (BODY): name "submit", value "Submit"
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Request body no files length: 149
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Input filter: Completed receiving request body (length 8893).
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Starting phase REQUEST_BODY.
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Hook insert_filter: Adding input forwarding filter (r 7f35a9d8d0a0).
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Hook insert_filter: Adding output filter (r 7f35a9d8d0a0).
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Input filter: Forwarding input: mode=0, block=0, nbytes=-1 (f 7f35a9d8e2b0, r 7f35a9d8d0a0).
[12/Sep/2013:15:12:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Input filter: Forwarded 8192 bytes.
[12/Sep/2013:15:13:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Initialising logging.
[12/Sep/2013:15:13:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Starting phase LOGGING.
[12/Sep/2013:15:13:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Recording persistent data took 0 microseconds.
[12/Sep/2013:15:13:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Audit log: Ignoring a non-relevant request.
[12/Sep/2013:15:13:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Multipart: Cleanup started (remove files 1).
[12/Sep/2013:15:13:48 +0300] [/sid#7f35a9f410a0][rid#7f35a9d8d0a0][/upload_file.php][4] Multipart: Deleted file (part) "/var/log/modsecurity_workdir/20130912-151248-AcAcATAcccAcAcRcvYAIpcAc-file-qmZcxo"

I can tentatively confirm that using nginx_refactoring I was able to fix this particular problem.

I, too, can confirm I'm running into this issue with nginx 1.6.2 where uploads over 8k with SecRequestBodyAccess On.
2015/02/02 11:11:00 [notice] 24627#0: ModSecurity for nginx (STABLE)/2.8.0 (http://www.modsecurity.org/) configured.
2015/02/02 11:11:00 [notice] 24627#0: ModSecurity: APR compiled version="1.3.9"; loaded version="1.3.9"
2015/02/02 11:11:00 [notice] 24627#0: ModSecurity: PCRE compiled version="7.8 "; loaded version="7.8 2008-09-05"
2015/02/02 11:11:00 [notice] 24627#0: ModSecurity: LIBXML compiled version="2.7.6"
I will have to test using the nginx_refactoring branch when I have time. For now, I have SecStatusEngine set to Off as file uploads are necessary.

Hi guys, few minutes ago I've merge #904 into nginx_refactoring branch. It should fix this issue, please confirm that the issue is fixed.

I just built the standalone ModSecurity module from the nginx_refactoring branch and compiled nginx 1.9.1 with it:
nginx version: nginx/1.9.1
built by gcc 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC)
built with OpenSSL 1.0.1k-fips 8 Jan 2015
TLS SNI support enabled
configure arguments: --add-module=../ModSecurity/nginx/modsecurity/ --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_spdy_module --with-cc-opt='-O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'

revelant portion of /etc/nginx/modsecurity.conf (~line 174)
SecDebugLog /var/log/modsecurity-debug.log
SecDebugLogLevel 6

Uploads >= 1MB are failing for me. nginx is forwarding these requests to an upstream server which handles the actual upload, FWIW.
I enabled the debugging log tried 2 uploads. 1 file < 1MB and another file equal to 1MB.
Please note I removed server information and the actual URI's in the following log snippets. Log for successful file upload:
[21/Jul/2015:16:47:34 +0000] Input filter: Forwarded 2063 bytes.
[21/Jul/2015:16:47:34 +0000]  Input filter: Sent EOS.
[21/Jul/2015:16:47:34 +0000] Input filter: Input forwarding complete.
[21/Jul/2015:16:47:34 +0000] Starting phase RESPONSE_HEADERS.
[21/Jul/2015:16:47:34 +0000] Output filter: Not buffering response body for unconfigured MIME type "application/json".
[21/Jul/2015:16:47:34 +0000] Output filter: Completed receiving response body (non-buffering).
[21/Jul/2015:16:47:34 +0000] Starting phase RESPONSE_BODY.
[21/Jul/2015:16:47:34 +0000] Output filter: Output forwarding complete.
[21/Jul/2015:16:47:34 +0000] Initialising logging.

Log for failing file upload:
[21/Jul/2015:17:01:02 +0000] Input filter: Forwarded 8192 bytes.
[21/Jul/2015:17:01:12 +0000] Initialising logging.
[21/Jul/2015:17:01:12 +0000] Starting phase LOGGING.

Disabling ModSecurity completely causes both file upload attempts to work.
Let me know if you want additional information and/or help testing.

I think it still exists in modsecurity2.9.1 for nginx, and I use nginx1.8.1 now.
The problem is that the function modsecurity_request_body_retrieve is wrongly used in function input_filter, if modsecurity_request_body_retrieve return 1, it means there are more chunks left, so it should be called again until it doesn't return 1.
It works fine now after I changed that.

how soon can you commit a fix ;)

Hi @mwang911,
Are you sure that you are inspecting a full request body after that modification? not only the first chunk?
I would like to suggest you guys to use the ModSecurity-nginx connector, available at: https://github.com/SpiderLabs/ModSecurity-nginx
Further information on libmodsecurity is available here: https://github.com/SpiderLabs/ModSecurity/tree/v3/master

Hi zimmerle,
I just test the file uploading and it works well now. It could retrieve all request body for nginx not only the max 8k. It is a advice I want you to consider, because I didn't make full other tests.  As far as i know, it is related to the retrieving of request body. The result is that nginx still think the request body has not been read.

@wellumies
Hi, the ngx_refactoring has done that in the right way. Look at the function input_filter in https://github.com/SpiderLabs/ModSecurity/blob/nginx_refactoring/apache2/apache2_io.c .
You can compare this function with the .tar.gz file for nginx in https://www.modsecurity.org/download.html.

No longer a concern in libModSecurity. Marking it as won't fix for 2.x.
Further information about libModSecurity available here:
https://github.com/SpiderLabs/ModSecurity/tree/v3/master

authors: applematt efx mwang911 victorhora wellumies zakarth zimmerle

repo: SpiderLabs/ModSecurity


text:
Remove enrichment options

This PR finalized simplifying enrichments, removing the enrichment_for and by options in favor of using multiple functions per enrichable message type (introduced in the related PR in core-java).
Notable changes:
Removed options and corresponding classes

enrichment_for and by options and support classes.
EnrichmentMessage is removed. Now any message can serve as an enrichment.  It could be entity state, for example, or standard message type.
EnrichmentType and related query methods were removed..
io.spine.type.enrichment package is removed.
Support for setting enrichment message interface is removed.

Other changes

Removed previously deprecated Logging.supplyFor().
MessageClass got method for querying super interfaces extending Message.
Classes implementing MessageContext no longer need to have a name ending with Context.
Elements of TaskName enumeration were renamed to their Gradle task counterparts.
Gradle-related testing utilities moved under io.spine.tools.gradle.testing package to avoid split-package problem with the main plugin JAR.
Introduced TaskSubject for testing Gradle tasks.

@armiol, @yuri-sergiichuk, PTAL. Most of the changes are removal.
This PR is the basis for this one from core-java.

source: gharchive/pull-request

repo: SpineEventEngine/base


text:
Get rid off dependency constants in version.gradle.kts in favor of default values passed to Spine object

We need to employ Kotlin feature of default parameter values instead of using constants defined under version.gradle.kts.
The current arrangement for handling those constants is cumbersome. It works, but it requires too much code, and it's prone to errors in usage.
It's time to make it simpler and easier to use.

@armiol, @yevhenii-nadtochii, FYI.

Closing as outdated.

repo: SpineEventEngine/config

license_source: github-api


text:
Server won't start on v 1521/711

The server wouldn't start even after I had removed all plugins/mods
http://pastebin.com/embed_js.php?i=Ns1BR5FN">

This was fixed just recently - try updating to the most recent version of Sponge

Unfortunately, the latest version cannot be downloaded at the moment
https://forums.spongepowered.org/t/latest-sponge-download-links/9588

authors: Billabonga ZephireNZ

repo: SpongePowered/Sponge

license: MIT


text:
Map scaling

When you currently scale the world in or out the camera perspective gets messed up: moves to the left & down. The more you scale the more left & down it goes.
This is because of the way the algorithm is written to make sure nothing outside the map is shown: so when the character comes to a border the camera doesn’t keep following him & shows the nothingness.
Easy fix is to have an alternate option for camera following: no algorithms during camera positioning calculation.
The nothingness outside the map will be visible but scaling won't be a problem anymore.
This way, one won't require new art assets, but simply will be able to scale the world by 2 while making sure texture filtering is set to nearest.

The map wasn't meant to be scaled so I see how this would mess up the auto follow logic. I will see if there is a work around. Also I suspect this will mess up the index for point logic too as the tileWidth/tileHeight will not be accurate after scaling. In the short term if you are targeting iOS 9 or higher I believe they added an SKCameraNode which might be worth looking into. I will close this as it isn't an issue but will look into this as a future feature.

repo: SpriteKitAlliance/SKAToolKit

license: mit


text:
[Add] Change the alert.

Details
I want to change the UI of alert
which is showing on failing log in.
Screenshots
Current:

Expected:

Type of Contribution

[ ] Update to an existing Animated Button
[ ] Adding a new Button
[ ] Resolving a bug
[X] Proposal to the Repository
[ ] Changes related to documentation or README.md
[ ] Other Changes

Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I am willing to work on this issue
[X] I am a GSSoC'23 contributor

@soubhik-111 We are using a different library.
So as considering our UI we had decided to use that toast.
Closing this as not required


text:
[FEATURE]: Readme Files Update

This is a(n):

[ ] Update to an existing Animated Button
[ ] Any Error
[ ] Proposal to the Repository
[x] Documentation / README.md changes

Details:

Hey, @Spyware007
I can help with making the readme file for this project. Please assign it to me.
PS: If you would like an updated template as well please raise an issue for that and assign it to me :)

@tuhinaww  Assigned to you!


text:
Add Translation section for README

Can we add a translation section within the README file ?

This section will provide translated versions of the content in different languages.

It would be a great idea for better accessibility and reach many people as possible.
If yes, kindly assign to me under GSSOC'23

Hello geoffreylgv, thanks for opening a issue, your contribution is valuable to us. The maintainers will review this issue and provide feedback as soon as possible.

Closing as not required

authors: SrijanShovit geoffreylgv shreya024


text:
RuntimeError: expected scalar type BFloat16 but found Float

Below is the log I have encountered at running "python scripts/txt2img.py --prompt "a professional photograph of an astronaut riding a horse" --ckpt <path/to/768model.ckpt/> --config configs/stable-diffusion/v2-inference-v.yaml --H 768 --W 768"
Running DDIM Sampling with 50 timesteps
DDIM Sampler:   0%|                                                                                | 0/50 [00:00<?, ?it/s]
data:   0%|                                                                                | 0/1 [00:00<?, ?it/s]
Sampling:   0%|                                                                                | 0/3 [00:00<?, ?it/s]
Traceback (most recent call last):
File "scripts/txt2img.py", line 388, in
main(opt)
File "scripts/txt2img.py", line 347, in main
samples, _ = sampler.sample(S=opt.steps,
File "/root/miniconda3/envs/ldmsd/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/mnt/disk1/swh/git_sd/stablediffusion/ldm/models/diffusion/ddim.py", line 104, in sample
samples, intermediates = self.ddim_sampling(conditioning, size,
File "/root/miniconda3/envs/ldmsd/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/mnt/disk1/swh/git_sd/stablediffusion/ldm/models/diffusion/ddim.py", line 164, in ddim_sampling
outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,
File "/root/miniconda3/envs/ldmsd/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/mnt/disk1/swh/git_sd/stablediffusion/ldm/models/diffusion/ddim.py", line 212, in p_sample_ddim
model_uncond, model_t = self.model.apply_model(x_in, t_in, c_in).chunk(2)
File "/mnt/disk1/swh/git_sd/stablediffusion/ldm/models/diffusion/ddpm.py", line 858, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "/root/miniconda3/envs/ldmsd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/disk1/swh/git_sd/stablediffusion/ldm/models/diffusion/ddpm.py", line 1335, in forward
out = self.diffusion_model(x, t, context=cc)
File "/root/miniconda3/envs/ldmsd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/disk1/swh/git_sd/stablediffusion/ldm/modules/diffusionmodules/openaimodel.py", line 797, in forward
h = module(h, emb, context)
File "/root/miniconda3/envs/ldmsd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/disk1/swh/git_sd/stablediffusion/ldm/modules/diffusionmodules/openaimodel.py", line 84, in forward
x = layer(x, context)
File "/root/miniconda3/envs/ldmsd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/disk1/swh/git_sd/stablediffusion/ldm/modules/attention.py", line 327, in forward
x = self.norm(x)
File "/root/miniconda3/envs/ldmsd/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/ldmsd/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 272, in forward
return F.group_norm(
File "/root/miniconda3/envs/ldmsd/lib/python3.8/site-packages/torch/nn/functional.py", line 2516, in group_norm
return torch.group_norm(input, num_groups, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: expected scalar type BFloat16 but found Float
Please, anyone has met the same and had a solution?

have you solved the issue?

have you solved the issue?

Yes I have. It is due to the incompatiblity of pytorch with cuda.

I am facing the same issue myself. Is it incompatible with cuda et al, or a version of it? because I have a hard time imagining running it without using the gpu. how did you fix it?

@picard314 I have run into this issue, but I was able to make adjustments so that the code runs, but it's using my CPU and not my NVIDIA GPU. I'm running CUDA 11.7 as that is what seemed to be the correct version. What CUDA version are you using, what all did you do to resolve this issue?

@wobblytables mine is cuda 11.4

Yes I have. It is due to the incompatiblity of pytorch with cuda

I had the same problem and solved it by setting up the gpu to run

Yes I have. It is due to the incompatiblity of pytorch with cuda

I had the same problem and solved it by setting up the gpu to run

I met with the same problem.Are you mean to use  methods like set CUDA_VISIBLE_DEVICES to set up the gpu?Thank you very much

@wobblytables mine is cuda 11.4
If for cuda 11.7, I think installation needs to be
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia

if you don't mind, can I know your GPU name and which version of pytorch you used?
I have geforce3060, and I used cuda 11.4, pytorch 1.12.1 but I met that error
so I changed the cuda version to 11.6 but still have a same problem...

python main.py --base=$cfg -t --gpus -1 --ckpt /mnt/cache/maqiang/pretrain/sd/512-base-ema.ckpt

Adding --device cuda worked for me.
It looks like a change setted CPU to be used by default https://github.com/Stability-AI/stablediffusion/pull/147/files#diff-048b7bba4049f97b2038502af5686b6c5f53a882ff02771fcb0d733d22a0ab6cR180-R186, I think it was messing up data types.

Adding --device cuda worked for me.
It looks like a change setted CPU to be used by default https://github.com/Stability-AI/stablediffusion/pull/147/files#diff-048b7bba4049f97b2038502af5686b6c5f53a882ff02771fcb0d733d22a0ab6cR180-R186, I think it was messing up data types.

nice solution,it's worked for me too

How do you fix this error when you actually want to run it on the CPU? I can't find a way to

you have to use a different config, in the path "configs/stable-diffusion" there is a folder called "intel" which can be used for running on the cpu, for example you can use -fp32 configs.

Is this going to get fixed?
I read the documentation, installed the requirements, and ran the example.  It crashed with this error message.
That seems like a pretty critical bug, but it hasn't even been assigned to anyone yet after 9 months.

as a hint, here is some description what might help: use "--precision full" (taken from here: https://huggingface.co/CompVis/stable-diffusion-v1-4/discussions/42) and in addition there are special configs for cpu processing in the "intel" folder of this repo. Currently I'm using the "-fp32" config in combination with the precision flag and it at least generates some images.
I'm not sure what the root-cause really is as I'm no expert in this field, but this https://github.com/Stability-AI/stablediffusion/blob/main/ldm/modules/attention.py#L175 looks suspicious...

i am having the same issue
!pip install torch==2.0.1 transformers datasets peft accelerate trl bitsandbytes optimum
when i try to load the X_IA3 adapters

authors: 320010ly Mateusmsouza SofiaBianchi123 adirz asdfjkluiop esiefker hotelbread lijain order-a-lemonade picard314 questor simonnxren wobblytables

repo: Stability-AI/stablediffusion


text:
Unable to  edit rule

Just installed version 1.5 using this: https://docs.stackstorm.com/install/rhel7.html
Created first Rule from UI. Now I'm unable to Edit it. When I press "Edit" button I get redirect to #/history

Problem is gone in 1.6. I guess some UI tests required for this project :)


text:
Regression of #3820 - Jinja is not rendered in action default values

ISSUE TYPE

Bug Report

STACKSTORM VERSION
# st2 --version
st2 2.7.2, on Python 2.7

OS / ENVIRONMENT / INSTALL METHOD
Install method = puppet-st2
OS = Red Hat Enterprise Linux Server release 7.5 (Maipo)

SUMMARY
array and object parameters are not being rendered in action metadata parameter defaults. integer parameters work fine.
STEPS TO REPRODUCE

1) create a new action metadata file
/opt/stackstorm/packs/default/actions/render_test.yaml
---
description: Run a local linux command
enabled: true
runner_type: mistral-v2
entry_point: workflows/render_test.yaml
name: render_test
pack: default
parameters:
  cmd:
    required: true
    type: string
  timeout:
    type: integer
    default: 60
  kv_array:
    type: array
    default: "{{ st2kv.system.kv_array | from_json_string }}"
  kv_object:
    type: object
    default: "{{ st2kv.system.kv_object | from_json_string }}"

2) create a new workflow
/opt/stackstorm/packs/default/actions/workflows/render_test.yaml
version: '2.0'

default.render_test:
  description: demo rendering failures
  type: direct
  input:
    - cmd
    - timeout
    - kv_array
    - kv_object

  tasks:
    task1:
      action: core.local
      input:
        cmd: "{{ _.cmd }}"

3) assign values in the datastore
$ st2 key set kv_array '["a", "b", "c"]'
$ st2 key set kv_object '{"a": "value", "b": "value2", "c": "value3"}'

4) register and run the action
$ st2 action create /opt/stackstorm/packs/default/actions/render_test.yaml
$ st2 run default.render_test cmd="ls"

EXPECTED RESULTS
Action to execute successfully with parameters:
cmd: ls
timeout: 60
kv_array:
  - a
  - b
  - c
kv_object:
  a: value
  b: value2
  c: value3

ACTUAL RESULTS
$ st2 run default.render_test cmd="ls"
ERROR: 400 Client Error: Bad Request
MESSAGE: '{{ st2kv.system.kv_array | from_json_string }}' is not of type 'array' for url: http://127.0.0.1:9101/executions

Error in /var/log/st2/st2api.log
2018-05-30 15:39:22,101 139814525031760 ERROR actionexecutions [-] Unable to execute action. Parameter validation failed.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/controllers/v1/actionexecutions.py", line 127, in _handle_schedule_execution
    pack=action_db.pack)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/controllers/v1/actionexecutions.py", line 186, in _schedule_execution
    liveaction_db, actionexecution_db = action_service.create_request(liveaction_db)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/services/action.py", line 89, in create_request
    allow_default_none=True)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/schema/__init__.py", line 294, in validate
    jsonschema.validate(instance=instance, schema=schema, cls=cls, *args, **kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/jsonschema/validators.py", line 541, in validate
    cls(schema, *args, **kwargs).validate(instance)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/jsonschema/validators.py", line 130, in validate
    raise error
ValidationError: u'{{ st2kv.system.kv_array | from_json_string }}' is not of type u'array'

Failed validating u'type' in schema['properties'][u'kv_array']:
    {u'default': u'{{ st2kv.system.kv_array | from_json_string }}',
     u'type': u'array'}

On instance[u'kv_array']:
    u'{{ st2kv.system.kv_array | from_json_string }}'
2018-05-30 15:39:22,103 139814525031760 ERROR router [-] Failed to call controller function "post" for operation "st2api.controllers.v1.actionexecutions:action_executions_controller.post": '{{ st2kv.system.kv_array | from_json_string }}' is not of type 'array'
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/router.py", line 470, in __call__
    resp = func(**kw)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/controllers/v1/actionexecutions.py", line 572, in post
    show_secrets=show_secrets)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/controllers/v1/actionexecutions.py", line 133, in _handle_schedule_execution
    abort(http_client.BAD_REQUEST, re.sub("u'([^']*)'", r"'\1'", e.message))
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/router.py", line 55, in abort
    raise exc.status_map[status_code](message)
HTTPBadRequest: '{{ st2kv.system.kv_array | from_json_string }}' is not of type 'array'

Just to confirm - did this behavior ever work in the past? If so, which version?

I believe it worked in the 2.6.x series. Just noticed it was broken in 2.7.2 (might have been broken sooner)

That's interesting since I don't remember us touching any of that code recently. Only somewhat related change was #4052
In any case, it looks like we should start with a test case. Another question also is why we don't have one for functionality we apparently support :)

I just tried to replicate the problem with the code you provided in v2.7.2, v2.7.1, v2.6.0 and v2.5.1. I get the same error message with every version (aka that functionality doesn't appear work / be supported).
Also looking at the PRs, that functionality was supposedly added in https://github.com/StackStorm/st2/pull/3892. Looking at the tests there - we only have tests for int scenario so likely there are more edge cases which are not handled correctly.

@Kami good to know, maybe i obviously never fully tested on my end.
Did a little digging this morning and found where it is failing specifically: https://github.com/StackStorm/st2/blob/master/st2common/st2common/util/param.py#L189
This is throwing the following exception:
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/controllers/v1/actionexecutions.py", line 127, in _handle_schedule_execution
    pack=action_db.pack)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/controllers/v1/actionexecutions.py", line 182, in _schedule_execution
    liveaction_db.context)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/param.py", line 306, in render_live_params
    context = _resolve_dependencies(G)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/param.py", line 217, in _resolve_dependencies
    context[name] = _render(node, context)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/param.py", line 196, in _render
    result = ENV.from_string(str(node['template'])).render(render_context)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 1, in top-level template code
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/jinja/filters/data.py", line 29, in from_json_string
    return json.loads(value)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

I turned on debug mode and here is the log for that node:
2018-05-31 07:20:58,097 139913045820592 INFO param [-] Rendering node: {'template': u'{{ st2kv.system.kv_array | from_json_string }}'} with context: {u'timeout': 60, 'st2kv': {'system': <st2common.services.keyvalues.KeyValueLookup object at 0x7f400b752a50>}}

@nmaludy Alright, after some more digging in it turns out it's not an actual issue in the code, but it's related to calling filter on the value which has already been de-serialized (aka filter is being called twice - once internally based on the action parameter definition and again inside the parameter Jinja string which is not needed).
You don't need to call from_json_string filter on the template value. This is done automatically based on the parameter type.
The following works fine for me:
---
description: Run a local linux command
enabled: true
runner_type: mistral-v2
entry_point: workflows/render_test.yaml
name: render_test
pack: default
parameters:
  cmd:
    required: true
    type: string
  timeout:
    type: integer
    default: 60
  kv_array:
    type: array
    default: "{{ st2kv.system.kv_array }}"
  kv_object:
    type: object
    default: "{{ st2kv.system.kv_object }}"

I will close that as not an issue.
Having said that, I do agree that the current exception is very unfriendly. At the very least, exception should include more data which would give user some clue what is going in (I will look into that change).

Awesome! I really swore i tested it...
I'll make a PR for st2docs and try to make another PR to add tests for objects and arrays

authors: Kami nmaludy


text:
st2 fails to store a Key-Value in the datastore if there is a "/" in the key name.

SUMMARY
st2 fails to store a Key-Value in the datastore if there is a "/" in the key name. I am sorry if I missed anything in the docs, which says that you cannot store a key with "/" in it and please feel free to close this issue if that is the case. Thank you.
STACKSTORM VERSION
st2 3.1.0, on Python 2.7.5
OS, environment, install method:
CentOS/Docker, custom install.
Steps to reproduce the problem
bash-4.2$ st2 key set "foo / bar" "foo_bar"
ERROR: 404 Client Error: Not Found
MESSAGE: The resource could not be found. for url: http://127.0.0.1:9101/v1/keys/foo%20/%20bar

Expected Results
st2 should have accepted the key-value pair and stored it.
Actual Results
st2 has failed to store the key-value pair.

Hi @armab I would like to take this up and send a fix.

I believe this issue probably lies in st2client / CLI and not the API itself (likely / character is not correctly URL encoded).
I believe we even have some st2api API level tests for keys with / in the name (and if we don't, we should add some).

Hi @Kami
I tried via the st2 command and also from the actions I have implemented, both self.action_service.set_value and self.action_service.get_value are failing with the same error. So I assume, it should be an API level issue.

It could be, although action service also utilizes st2client code which talks to the API :)
If there is not an existing test already, I would start with an API level test for that functionality.

@Kami @armab I see no tests which stores a key-value pair with a "/" in the key name.
Also, I am not able to run the tests in test_kvps.py file using the following command from the root directory as mentioned in this doc on a Ubuntu machine:
nosetests --nocapture ./st2api/tests/unit/controllers/v1/test_kvps.py
It says ERROR: Failure: ImportError (No module named st2tests.api)
I tried exporting/setting PYTHONPATH to st2 directory but no luck. Could you please help me to get the tests running ? I tried with both Python2 and Python3.
Thanks.

For developing StackStorm platform itself, I would recommend you to use this Vagrant image - https://github.com/StackStorm/st2vagrantdev
In short, you need to run make requirements which will create virtualenv, install all the dependencies and set PYTHONPATH correctly.

@Kami Thanks. This is Great.
I am able to setup the vagrant environment and I can run the tests now. Also, I have added a test which stores a KV and there is a "/" in the key name and it fails. I will dig through the code and see, if we can encode "/" someway.
@armab

@Kami @armab
I modified the serialize method here to quote (from six.moves.urllib_parse import quote) so that the key for example a/b will be converted into a%2Fb. However, the st2 key set "a/b" "some value" is failing with resource could not be found error.
ERROR: 404 Client Error: Not Found
MESSAGE: The resource could not be found. for url: http://127.0.0.1:9101/v1/keys/a%2Fb

I think, I will have to make some modifications on the API as well to accept this kind of keys?
Thank you.

Hi @Kami @armab Do you think, we should be able to encode and store keys with /? or I will try to implement something which will thrown an error if the key has / in it. Thank you.

Yes, I think it's absolutely reasonable to be able to use the / in Key name.
Quick example: https://www.consul.io/docs/commands/kv/put.html#examples
If you could make it work and support your enhancement with tests as well, - that would be great addition :+1:

Thanks for the reply @armab
As I mention in the previous comment, I can encode the key on the st2client side and I can see the  PUT call is also on the encoded key like below:
(virtualenv) vagrant@ubuntu-xenial:~/local/st2$ st2 key set foo/bar foo
# -------- begin 140367719226128 request ----------
curl -X PUT -H  'Connection: keep-alive' -H  'Accept-Encoding: gzip, deflate' -H  'Accept: */*' -H  'User-Agent: python-requests/2.23.0' -H  'content-type: application/json' -H  'Content-Length: 70' --data-binary '{"scope": "system", "name": "foo%2Fbar", "value": "foo", "user": null}' http://127.0.0.1:9101/v1/keys/foo%2Fbar
# -------- begin 140367719226128 response ----------
{
    "faultstring": "The resource could not be found."
}
# -------- end 140367719226128 response ------------

ERROR: 404 Client Error: Not Found
MESSAGE: The resource could not be found. for url: http://127.0.0.1:9101/v1/keys/foo%2Fbar

I am trying to find the code which is accepting this request and make it work for encoded / or am I missing something?
Any help in making me understand the api side of things would be appreciated. Thank you.
@Kami

@Kami @armab Where can I find the API side related code? As you can see in my previous comment, API isn't consider a key with encoded / as a new key in the PUT call.
Could you please direct me to some documentation or any other reference/example? Thank you.

Start with the st2 api KeyValue controller in: https://github.com/StackStorm/st2/blob/master/st2api/st2api/controllers/v1/keyvalue.py
For development environment, standards and expectations, check the https://docs.stackstorm.com/development/index.html

@armab @Kami It looks like if I encode a key foo/bar to foo%2Fbar on the client side, the call which will be made will look like this:
curl -X PUT -H  'Connection: keep-alive' -H  'Accept-Encoding: gzip, deflate' -H  'Accept: */*' -H  'User-Agent: python-requests/2.23.0' -H  'content-type: application/json' -H  'Content-Length: 69' --data-binary '{"scope": "system", "name": "foo%2Fbar", "value": "f1", "user": null}' http://127.0.0.1:9101/v1/keys/foo%2Fbar

But on the api side, it is failing while trying to match the requests path to a controller here.
For some reason, the req.path is going back to foo/bar instead of being the encodedfoo%2Fbar. As you can see, this happening before entering the code in https://github.com/StackStorm/st2/blob/master/st2api/st2api/controllers/v1/keyvalue.py
So I am guessing somewhere else, we need to make change to persist the encoded key.

I did this little test based on the code in router.py:

import webob.compat
path="http://127.0.0.1:9101/v1/keys/foo%2Fbar"
webob.compat.url_unquote(path)
'http://127.0.0.1:9101/v1/keys/foo/bar'

So it looks like it gets altered to foo/bar by the url_unquote call.

Thank you for the input @amanda11.
I just tried with double encoding/quoting the key name if it contains a / while I set or get the KV pair and it is working as expected. If this OK, I will go ahead and submit a pull request.
@Kami @armab
git diff:
diff --git a/st2client/st2client/commands/keyvalue.py b/st2client/st2client/commands/keyvalue.py
index 8eed47364..d1a2305eb 100644
--- a/st2client/st2client/commands/keyvalue.py
+++ b/st2client/st2client/commands/keyvalue.py
@@ -21,6 +21,7 @@ import logging
 from os.path import join as pjoin

 import six
+from six.moves.urllib_parse import quote

 from st2client.commands import resource
 from st2client.commands.noop import NoopCommand
@@ -141,6 +142,8 @@ class KeyValuePairGetCommand(resource.ResourceGetCommand):
     @resource.add_auth_token_to_kwargs_from_cli
     def run(self, args, **kwargs):
         resource_name = getattr(args, self.pk_argument_name, None)
+        if '/' in resource_name:
+            resource_name = quote(quote(resource_name, safe=''))
         decrypt = getattr(args, 'decrypt', False)
         scope = getattr(args, 'scope', DEFAULT_GET_SCOPE)
         kwargs['params'] = {'decrypt': str(decrypt).lower()}
@@ -185,8 +188,14 @@ class KeyValuePairSetCommand(resource.ResourceCommand):
     @resource.add_auth_token_to_kwargs_from_cli
     def run(self, args, **kwargs):
         instance = KeyValuePair()
-        instance.id = args.name  # TODO: refactor and get rid of id
-        instance.name = args.name
+        key_name = args.name
+        # urllib_parse.quote the key name to support keys with '/' in them.
+        # We double quote it here, as it will unquoted once on the API side.
+        if '/' in key_name:
+            key_name = quote(quote(args.name, safe=''))
+
+        instance.id = key_name  # TODO: refactor and get rid of id
+        instance.name = key_name
         instance.value = args.value
         instance.scope = args.scope
         instance.user = args.user

A PR to fix this issue which is still around is very welcome, @RaviTezu :)

authors: Kami RaviTezu amanda11 armab winem


text:
st2cd KVStore Compatibility >=0.9

This PR attempts to load up the old and new libraries to access the K/V store, allowing compatibility with older running nodes (<0.8 ) and newer nodes (>=0.9)
/cc @DoriftoShoes

Tested and working on st2ops001

+1

authors: DoriftoShoes jfryman


text:
Update dependencies

And pin ThingTalk to the 1.1.0 branch, because the next version
is likely to include dangerous stuff that should go into Cloud Almond
1.1 at least.

Yeah no, dep updates should be handled differently (Greenkeeper maybe?)


text:
Pennant Exercising a DMA Bug

Pennant (without control replication) is exercising a bug in the new DMA system in the master branch.  Below is a command line that runs correctly in the 'olddma' branch, but results in bad data (a pointer in this case) being read from a field in the 'master' branch.  Notice that command line uses '-lg:inorder' to ensure that the runtime is doing things in strict program order.  Also notice the '-fbounds-checks' means this code takes a while to run so be patient.
Build regent with: ./install.py --debug --gasnet
LAUNCHER='mpirun -H n0002,n0003 -np 4 --map-by numa:pe=6 -x LD_LIBRARY_PATH -x INCLUDE_PATH -x TERRA_PATH --oversubscribe' ./regent.py examples/pennant_fast.rg examples/pennant.tests/leblanc/leblanc.pnt -ll:cpu 5 -npieces 20 -numpcx 1 -numpcy 20  -par_init 1 -seq_init 0 -fbounds-checks 1 -lg:inorder
When the code fails in the 'master' branch you will see the following error:
Errors reported during runtime.
examples/pennant_fast.rg:122: pointer ptr(zone(), $rz) is out-of-bounds
@magnatelee first reported this bug so ask him to test it when there is a fix.

By commenting out the application assertions I was able to validate the runtime execution with Legion Spy, so I'm now confident that the issue is in the DMA system.
I've also noticed other weird failure modes where it is not just the pointer data that is invalid, suggesting that copies of any field are subject to having their data corrupted.

I'm about to assign the bug back to Sean and Zhihao, but before I do here is a summary of what we learned and why we now know the bug is in the DMA system (both new AND old have the same bug).  We've simplified the test case to a single node and it can be reproduced with the following command line:
./regent.py examples/pennant_fast.rg examples/pennant.tests/leblanc/leblanc.pnt -ll:cpu 2 -npieces 2 -numpcx 1 -numpcy 2 -par_init 1 -seq_init 0 -lg:inorder -fbounds-checks 1

We know this is the commit that causes the problem: 390235e  It changes the sizes of the physical instances being used.
The same behavior can be observed with both the master branch and the olddma branch, so it is nothing related to the new DMA system specifically.
The application is passing with -fbounds-checks 1 so there is no corruption of the data by the application itself.
Legion Spy validates this execution as being correct by the Legion runtime and @TWarszawski and I have hand-checked about 75% of the graph ourselves.
The first task to fail is the calc_volumes_full task during the start-up and it fails the 'sv > 0.0' assertion.
The first cell to fail is cell at location 1768 in the first (and only span).
The reason it fails is that (deterministically) different data is being read from the fields 'zv.x' and 'zv.y' during the computation of the 'sa' temporary.  Specifically we see the following values for the computation of 'sa' in the 'good' and 'bad' cases:

sa good: 0.0025 = 0.5 * cross(<0.3,4.4> - <0.2,4.4>,<0.25,4.45> - <0.2,4.4>)
sa bad: -0.05375 = 0.5 * cross(<0.3,4.4> - <0.2,4.4>,<0.175,3.325> - <0.2,4.4>)

If we examine the event graph for a bad run we see that the data for this field is placed in a concrete instance and never moved.  No copies to or from this instance are issued by the runtime before the failure, which is correct.  The proper event dependences ensure that all tasks associated with this field are running in the correct order which is consistent with Legion Spy validation.
I traced the bad value back to 'calc_centers_full' where I determined that the difference comes from the computation of 'zx'.  On the third iteration for zone 442 the good and the bad differ while still having the same pointers for p1.

Good:
p1_px= <0.2,4.4> p1=497
p1_px= <0.3,4.4> p1=498
p1_px= <0.3,4.5> p1=3
p1_px= <0.2,4.5> p1=2
znump 4
zx 442 is <0.25,4.45> nside=4
Bad:
p1_px=<0.2,4.4> p1=497
p1_px=<0.3,4.4> p1=498
p1_px=<0,0> p1=3
p1_px=<0.2,4.5> p1=2
znump 4
zx 442 is <0.175,3.325> nside=4

The p1.px field is initialized by 'intiialize_topology' so I went and printed out the values that are computed for the 'px' field for location p1=3 for both the good and bad versions:

good: p1=3 = <0.3,4.5> Shared Bottom
bad: p1=3 = <0.3,4.5> Shared Bottom
Good news! They're the same, that means that something is going wrong between the tasks.

If you look at the attached event graph you will see that there is a copy needed between the instances used by 'initialize_topology' and 'calc_centers_full'.  Legion correctly issues this copy (Realm copy 19).  Legion Spy indicates that for this particular execution both index spaces 10 and 12 are exactly the same, so their intersection is the same (the copy is an intersection of index spaces 10 and 12) so all of the data should be moved.

Reducing top-level index space shapes...
Done
Space Index Space 10 has 3 points
Points: 3 7 6
Space Index Space 12 has 3 points
Points: 3 7 6
However, clearly, p1=3 is not being moved properly, and that is a DMA system bug.
bad.pdf

Note that Legion is doing the intersection computation using Realm primitives so you might want to check this code too:
https://github.com/StanfordLegion/legion/blob/master/runtime/legion/region_tree.cc#L5177-L5188

I've managed to reproduce the error, and am digging into it now.

I'm still sorting through some stuff, but there is only a single copy that occurs before calc_centers_full reads the bad value, and it appears to be copying an indexspace that is the non-empty intersection of two index spaces into an instance that was created from an empty index space.  Realm does not verify that the domain used for a copy is a subset of the domains that exist in source and destination instances (because it is too expensive), but if there were such a check, it'd be complaining loudly here.
Some snippets from the log:
[0 - b0910000] {2}{region}: subregion 500000000000000a (of 5000000000000007) restricted to [0,10]
[0 - b0910000] {2}{meta}: index space created: id=500000000000000a parent=5000000000000007 (num_elmts=1001)
[0 - b0910000] {2}{region}: subregion 500000000000000c (of 5000000000000007) restricted to [0,10]
[0 - b0910000] {2}{meta}: index space created: id=500000000000000c parent=5000000000000007 (num_elmts=1001)
[0 - b0910000] {2}{region}: subregion 500000000000000d (of 5000000000000007) restricted to [-1,-1]
[0 - b0910000] {2}{meta}: index space created: id=500000000000000d parent=5000000000000007 (num_elmts=1001)

[0 - b0d16000] {2}{inst}: local instance 6000000000000007 created in memory 1e00000000000000 at offset 536775424+72 (redop=0 list_size=-1 parent_inst=0 block_size=4)
[0 - b0d16000] {2}{meta}: instance created: region=500000000000000d memory=1e00000000000000 id=6000000000000007 bytes=72

[0 - b050a000] {3}{index_spaces}: creating intersection: 500000000000000a & 500000000000000c
[0 - b050a000] {2}{region}: subregion 5000000000000026 (of 0) restricted to [0,10]
[0 - b050a000] {2}{meta}: index space created: id=5000000000000026 num_elmts=64
[0 - b050a000] {1}{dma}: copy: 1 distinct src/dst mem pairs, is=5000000000000026
[0 - b050a000] {2}{dma}: dma request 0x7fb3441ecc40 created - is=5000000000000026 before=0 after=8000000002c00008
[0 - b050a000] {2}{dma}: dma request 0x7fb3441ecc40 field: 6000000000000002[0]->6000000000000007[0] size=8 serdez=0
[0 - b050a000] {2}{dma}: dma request 0x7fb3441ecc40 field: 6000000000000002[8]->6000000000000007[8] size=8 serdez=0

@streichler Are you sure you got the right point for calc_centers_full (there are two of them) and the right copy?  I think the copy we're interested in here should be from 6000000000000002 to 600000000000000b and not 6000000000000007.  If you're running with -lg:inorder all the mapping decisions should be deterministic and therefore the instance creation names too.  At least they were for me when I was debugging.  Instance 600000000000000b should be an instance of index space 10 which is non-empty.

This was the only copy to have completed before the first calc_centers_full executed.  (I modified the test to assert on bad data in there, so the other instance of calc_centers_full never executes either.)  The only other copy that has even been requested at that point is dependent on the first calc_centers_full instance's completion, and appears to target the same empty instance.

Reassigning this to @lightsighter .  There's definitely something wrong in the instance creation.  Here's a reformatted excerpt of legion_spy.py -i on my trimmed down test:
Instance 0x6000000000000000  Region (4,1,1)
Instance 0x6000000000000001  Region (8,2,2)
Instance 0x6000000000000002  Region (12,2,2)
Instance 0x6000000000000003  Region (10,2,2)
Instance 0x6000000000000004  Region (14,3,3)
Instance 0x6000000000000005  Region (5,1,1)
Instance 0x6000000000000006  Region (9,2,2)
Instance 0x6000000000000007  rp_all (2,2,2)
Instance 0x6000000000000008  Region (11,2,2)
Instance 0x6000000000000009  Region (15,3,3)
Instance 0x600000000000000a  Region (18,4,4)
Instance 0x600000000000000b  Region (22,5,5)
Instance 0x600000000000000c  Region (26,6,6)
Instance 0x600000000000000d  Region (30,7,7)
Instance 0x600000000000000e  Region (19,4,4)
Instance 0x600000000000000f  Region (23,5,5)
Instance 0x6000000000000010  Region (27,6,6)
Instance 0x6000000000000011  Region (31,7,7)
Instance 0x6000000000000012  Region (4,1,1)
Instance 0x6000000000000013  Region (5,1,1)
Instance 0x6000000000000014  Region (14,3,3)

And here's the Realm logging of instance creations:
[0 - b0284000] {2}{meta}: instance created: region=5000000000000004 memory=1e00000000000000 id=6000000000000000 bytes=452
[0 - b0284000] {2}{meta}: instance created: region=5000000000000008 memory=1e00000000000000 id=6000000000000001 bytes=8928
[0 - b0284000] {2}{meta}: instance created: region=500000000000000c memory=1e00000000000000 id=6000000000000002 bytes=216
[0 - b0284000] {2}{meta}: instance created: region=500000000000000a memory=1e00000000000000 id=6000000000000003 bytes=192
[0 - b0284000] {2}{meta}: instance created: region=500000000000000e memory=1e00000000000000 id=6000000000000004 bytes=75600
[0 - b0d16000] {2}{meta}: instance created: region=5000000000000005 memory=1e00000000000000 id=6000000000000005 bytes=452
[0 - b0d16000] {2}{meta}: instance created: region=5000000000000009 memory=1e00000000000000 id=6000000000000006 bytes=8928
[0 - b0d16000] {2}{meta}: instance created: region=500000000000000d memory=1e00000000000000 id=6000000000000007 bytes=72
[0 - b0d16000] {2}{meta}: instance created: region=500000000000000b memory=1e00000000000000 id=6000000000000008 bytes=192
[0 - b0d16000] {2}{meta}: instance created: region=500000000000000f memory=1e00000000000000 id=6000000000000009 bytes=75600
[0 - b111c000] {2}{meta}: instance created: region=5000000000000011 memory=1e00000000000000 id=600000000000000a bytes=68
[0 - b111c000] {2}{meta}: instance created: region=5000000000000014 memory=1e00000000000000 id=600000000000000b bytes=68
[0 - b111c000] {2}{meta}: instance created: region=5000000000000017 memory=1e00000000000000 id=600000000000000c bytes=68
[0 - b111c000] {2}{meta}: instance created: region=500000000000001a memory=1e00000000000000 id=600000000000000d bytes=68
[0 - b0284000] {2}{meta}: instance created: region=5000000000000012 memory=1e00000000000000 id=600000000000000e bytes=68
[0 - b0284000] {2}{meta}: instance created: region=5000000000000015 memory=1e00000000000000 id=600000000000000f bytes=68
[0 - b0284000] {2}{meta}: instance created: region=5000000000000018 memory=1e00000000000000 id=6000000000000010 bytes=68
[0 - b0284000] {2}{meta}: instance created: region=500000000000001b memory=1e00000000000000 id=6000000000000011 bytes=68
[0 - b0284000] {2}{meta}: instance created: region=5000000000000004 memory=1e00000000000000 id=6000000000000012 bytes=14464
[0 - b131f000] {2}{meta}: instance created: region=5000000000000005 memory=1e00000000000000 id=6000000000000013 bytes=14464
[0 - b0284000] {2}{meta}: instance created: region=500000000000000e memory=1e00000000000000 id=6000000000000014 bytes=28800

It looks like the Legion and Realm index space numbers deviate after a while, but all of the first ones match up EXCEPT for instance 6000000000000007, which happens to be the target of the copy, and for which that copy would make a lot more sense if the instance had been constructed from rp_all instead of (13,2,2) (which looks like it's rp_shared[1] from the index space tree).

I believe that this bug is now fixed with da5e088. Assigning back to @magnatelee to confirm.

The bug seems fixed. Close this issue.

authors: lightsighter magnatelee streichler

repo: StanfordLegion/legion


text:
To load parquet file has been fail.

We've tried to load parquet file from s3 api.
(A table with 8,000 columns has roughly 5,000 rows.)
Few hour later, some starrocks BE nodes have killed by self.
But memory and storage space was enough to insert records.
Steps to reproduce the behavior (Required)

CREATE TABLE '...'
column#: 8,000 (2000 integer, 6000 decimal)

CREATE TABLE tbl_pk_dec_c8000_r200000
(
    int_col1 INT,
    int_col2 INT,
    int_col3 INT,
    int_col4 INT,
    int_col5 INT,
...
    dec_col5999 DECIMAL(17,7),
    dec_col6000 DECIMAL(17,7)
)
PRIMARY KEY(int_col1)
DISTRIBUTED BY HASH(int_col1) BUCKETS 20
PROPERTIES("replication_num" = "3");

INSERT INTO '....'

LOAD LABEL tbl_pk_dec_c8000_r200000
(
    DATA INFILE("s3a://test/dec_c8000_r200000.001.parquet")
    INTO TABLE tbl_pk_dec_c8000_r200000
    -- COLUMNS TERMINATED BY ","
    FORMAT AS "PARQUET"
    (
    int_col1,
    int_col2,
    int_col3,
...
    dec_col5998,
    dec_col5999,
    dec_col6000
    )
)
WITH BROKER
(
    "aws.s3.endpoint" = "<storage_addr>",
    "aws.s3.access_key" = "....",
    "aws.s3.secret_key" = "....",
    "aws.s3.enable_ssl" = "true",
    "aws.s3.enable_path_style_access" = "true"
);

dec_c8000_r200000.001.parquet

size: 1.1G
rows: 5,115

Expected behavior (Required)
complete to load
Real behavior (Required)
memory status (top)

show load\G
*************************** 2. row ***************************
         JobId: 14005
         Label: tbl_pk_dec_c8000_r200000
         State: CANCELLED
      Progress: ETL:N/A; LOAD:N/A
          Type: BROKER
      Priority: NORMAL
      ScanRows: 835
  FilteredRows: 0
UnselectedRows: 0
      SinkRows: 835
       EtlInfo: NULL
      TaskInfo: resource:N/A; timeout(s):14400; max_filter_ratio:0.0
      ErrorMsg: type:LOAD_RUN_FAIL; msg:[E1014]Got EOF of Socket{id=116 fd=190 addr=10.202.14.125:8060:50134} (0x0x7ffab6563740) [R1][E112]Not connected to 10.202.14.125:8060 yet, server_id=116 [R2][E112]Not connected to 10.202.14.125:8060 yet, server_id=116 [R3][E112]Not connected to 10.202.14.125:8060 yet, server_id=116
    CreateTime: 2023-09-25 14:13:17
  EtlStartTime: 2023-09-25 14:13:23
 EtlFinishTime: 2023-09-25 14:13:23
 LoadStartTime: 2023-09-25 14:13:23
LoadFinishTime: 2023-09-25 14:56:48
   TrackingSQL:
    JobDetails: {"All backends":{"d2f70bc9-a641-442d-a997-9bb0f71cade0":[10009,10007,10006,10008]},"FileNumber":1,"FileSize":1155971344,"InternalTableLoadBytes":53505686,"InternalTableLoadRows":835,"ScanBytes":53441670,"ScanRows":835,"TaskNumber":1,"Unfinished backends":{"d2f70bc9-a641-442d-a997-9bb0f71cade0":[10009,10007,10006]}}

be2: be.WARNING
W0925 14:56:44.756848 404252 stack_util.cpp:350] 2023-09-25 14:56:44.756733, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
     @          0xbd4c46d  __wrap___cxa_throw
    @          0x863f966  _Znwm.cold
    @          0x86d9a4c  std::__new_allocator<>::allocate()
    @          0x86d6aff  std::allocator_traits<>::allocate()
    @          0x8743d0c  std::_Vector_base<>::_M_allocate()
    @          0x8743bab  std::_Vector_base<>::_M_create_storage()
    @          0x874195b  std::_Vector_base<>::_Vector_base()
    @          0x8831a4d  std::vector<>::vector()
    @          0x88102a3  starrocks::FixedLengthColumnBase<>::FixedLengthColumnBase()
    @          0x8bb4919  starrocks::ColumnFactory<>::ColumnFactory<>()
    @          0x8bb46f1  starrocks::FixedLengthColumn<>::FixedLengthColumn()
    @          0xc48a5cb  std::_Construct<>()
    @          0xc489d90  std::allocator_traits<>::construct<>()
    @          0xc48912b  std::_Sp_counted_ptr_inplace<>::_Sp_counted_ptr_inplace<>()
    @          0xc487ba4  std::__shared_count<>::__shared_count<>()
    @          0xc486930  std::__shared_ptr<>::__shared_ptr<>()
    @          0xc485d67  std::shared_ptr<>::shared_ptr<>()
    @          0xc485502  std::make_shared<>()
    @          0xc484e1a  starrocks::ColumnFactory<>::create<>()
    @          0xc48418b  starrocks::ColumnBuilder::operator()<>()
    @          0xc4834f3  _ZN9starrocks20type_dispatch_columnINS_13ColumnBuilderEJNS_14TypeDescriptorEmEEEDaNS_11LogicalTypeET_DpT0_
    @          0xc481a51  starrocks::ColumnHelper::create_column()
    @          0xca1caf9  starrocks::serde::ProtobufChunkDeserializer::deserialize()
    @          0xbb3a082  starrocks::LoadChannel::_deserialize_chunk()
    @          0xbb38d2f  starrocks::LoadChannel::add_chunk()
    @          0xbb31f04  starrocks::LoadChannelMgr::add_chunk()
    @          0xbc6049b  starrocks::BackendInternalServiceImpl<>::tablet_writer_add_chunk()
    @          0xc1df97a  doris::PBackendService::CallMethod()
    @          0xd44f1a1  brpc::policy::ProcessRpcRequest()
    @          0xd533707  brpc::ProcessInputMessage()
    @          0xd5345a7  brpc::InputMessenger::OnNewMessages()
    @          0xd3dac1d  brpc::Socket::ProcessEvent()

StarRocks version (Required)

You can get the StarRocks version by executing SQL select current_version()

3.1.2-4f3a2ee91b

Is /proc/sys/vm/overcommit_memory turned on?

@meegoo
yes. It has been turned on.

Is it possible to also try the files() function? https://docs.starrocks.io/en-us/latest/sql-reference/sql-functions/table-functions/files

Sorry. I am too late.
I've tried to your suggestion as below.
But, it didn't work.
--  ___test_file.sql
CREATE TABLE dec_c8000_r200000_001 AS
    SELECT * FROM FILES(
        "path" = "s3://test/dec_c8000_r200000.001.parquet",
        "format" = "parquet",
        "aws.s3.access_key" = "******",
        "aws.s3.secret_key" = "******",
        "aws.s3.region" = "us-west-2"
);

result
mysql> source ___test_file.sql;
ERROR 1064 (HY000): Access storage error. Unknown error
mysql>

You'll have to break the parquet file to be smaller or use the upcoming PIPE feature to load data.

authors: alberttwong kakao-lunarvel-vet meegoo

repo: StarRocks/starrocks


text:
"select array_concat(array,empty array) from table"  is different from the expected result

Steps to reproduce the behavior (Required)

CREATE TABLE `array_sort_null` ( c1 int NOT NULL, c2 array<date>, c3 array<datetime>, c4 array<char(20)>, c5 array<varchar(20)>, c6 array<boolean>, c7 array<tinyint>, c8 array<smallint>, c9 array<int>, c10 array<bigint>, c11 array<largeint>, c12 array<float>, c13 array<double> ) DUPLICATE KEY(c1) DISTRIBUTED BY HASH(c1) buckets 1 PROPERTIES ("replication_num"="1");

insert into array_sort_null (c1, c2, c3, c4, c5, c6, c7, c8, c9, c10, c11, c12, c13)values (1, ['2012-01-02', '2012-01-01', '2012-01-04'], ['2012-01-04 01:01:01', '2012-01-03 01:01:01', '2013-01-01 01:01:01'], ['22', '33', '11'], ['222', null, '111'], [False, True, False], [2, 1, 3], [22, 11, 33], [null, null, null], [null, null, null], [3333, 4231, 1111], [11.2, 11.1, 10.1], [11.11, 8.99, 4.44]), (2, ['2013-01-02', '2013-01-01', '2014-01-04'], ['2013-01-04 01:01:01', '2013-01-03 01:01:01', '2010-01-01 01:01:01'], ['22', '33', '11', '66'], ['222', null, null, '000'], [False, True, False], [2, 4, 3], [22, 11, 33], [222, 111, 333], [null, null, null], [3343, 4831, 1711], [11.3, 11.2, 10.2], [11.22, 8.77, 4.33]);

select array_concat(["2012-01-02","2012-01-01","2012-01-04"],[]) from array_sort_null where c1=1;

Expected behavior (Required)
["2012-01-02","2012-01-01","2012-01-04"]
Real behavior (Required)
ERROR 1064 (HY000): Index: 0, Size: 0
StarRocks version (Required)
+------------------------------------------+
| current_version()                        |
+------------------------------------------+
| PIPELINE_SQLANCER_MASTER_RELEASE b61b70b |
+------------------------------------------+

It can't re-produce. It should be fixed.

authors: Pslydhh lvchenyang-maker


text:
[Feature]filetable with recursive path

Signed-off-by: zombee0 flylucas_10@163.com
What type of PR is this：

[ ] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Which issues of this PR fixes ：

Fixes #
Problem Summary(Required) ：

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[ ] I have checked the version labels which the pr will be auto backported to target branch

[ ] 2.5
[ ] 2.4
[ ] 2.3
[ ] 2.2

[FE PR Coverage Check]
:disappointed: fail : 0 / 28 (00.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/planner/FileTableScanNode.java
0
1
00.00%
[80]

:large_blue_circle:
com/starrocks/catalog/FileTable.java
0
2
00.00%
[109, 111]

:large_blue_circle:
com/starrocks/connector/hive/HiveRemoteFileIO.java
0
25
00.00%
[104, 105, 106, 108, 111, 112, 114, 118, 119, 121, 123, 124, 125, 126, 128, 130, 131, 132, 133, 134, 135, 136, 137, 138, 140]


text:
How can I set match files to specific file types?

I found that the default setting is /*, but I just want to match .sv and .v files.
I've tried the
**/{.sv,.v}//*
and
/{.sv,.v}//*
as the exclude setting style, not working.

Hello! This should already be possible by changing the commentAnchors.workspace.matchFiles config property to **/*.{sv,v}
Currently this only applies to the workspace anchors panel. The next release will apply the same behavior to opened text files.

repo: StarlaneStudios/vscode-comment-anchors


text:
Proposal of tweaks around global and local cache configuration

Making sure global cache configuration is taken into consideration for all datasource methods
Making sure method-specific cache configuration is overriding global configuration, if present
Adding related tests
Adding simple VSC config to enforce basic code styling
Adding VSC launcher to help out with debugging tests and code

@StarpTech I'm happy to talk through the changes I've made and the motivation, I also have following changes queued up, ie. for optional maxTtlIfError and I'm thinking on improving performance and bandwidth used while caching with maxTtlIfError

Hi @kdybicz sorry for the late response. Your change will give request options precedence over global defaults. This is natural and the current behavior was more of a bug.

authors: StarpTech kdybicz

repo: StarpTech/apollo-datasource-http


text:
接收到图片后没有返回

环境py3.9.12、nb2rc4，自建仓库，插件使用最新版本，没有报错，网络可以正常访问抱脸

检查了一下，发现是huggingface的推断接口API地址变更导致的
比如，默认的地址更改前是：https://hf.space/embed/ppxxxg22/Real-ESRGAN/api/predict/，更改后应该为：https://ppxxxg22-real-esrgan.hf.space/api/predict/
如果是其他自己建的仓，API仿照第二个格式进行更改即可。

检查了一下，发现是huggingface的推断接口API地址变更导致的
比如，默认的地址更改前是：https://hf.space/embed/ppxxxg22/Real-ESRGAN/api/predict/，更改后应该为：https://ppxxxg22-real-esrgan.hf.space/api/predict/
如果是其他自己建的仓，API仿照第二个格式进行更改即可。

感谢，辛苦了

检查了一下，发现是huggingface的推断接口API地址变更导致的
比如，默认的地址更改前是：https://hf.space/embed/ppxxxg22/Real-ESRGAN/api/predict/，更改后应该为：https://ppxxxg22-real-esrgan.hf.space/api/predict/
如果是其他自己建的仓，API仿照第二个格式进行更改即可。

替换后可以了

authors: ElainaFanBoy Staskaer


text:
Decide on the usage of .ConfigureAwait(false) in Steeltoe code

Today the Steeltoe codebase contains limited usage of .ConfigureAwait(false). I'm wondering if that was a deliberate decision, and if so, what the rationale was for sometimes using it and sometimes omitting it. Should we revisit this, now that we're not targeting .NET Framework and NetStandard anymore?
There's quite some contradicting guidance around the topic.

The blog at https://itnext.io/a-deep-dive-into-configureawait-65f52b9605c2 states that there's no more need for it when targeting only .NET Core:

At this moment, one might think that in .NET Core you won’t need to spread ConfigureAwait(false) all over your code. Almost!
This is almost true, it is still recommended the utilization of ConfigureAwait(false) for libraries as a fallback if those libraries are used within a legacy framework. But for most of the cases yes, in .NET Core you can drop the ConfigureAwait(false) usage.

EF Core recently switched to adding .ConfigureAwait(false) almost everywhere, despite requiring .NET Core.

The ASP.NET Core repo removed .ConfigureAwait(false) from all projects that don't target NetStandard.

Excerpt from the ConfigureAwait FAQ by Stephen Toub:

I’ve heard ConfigureAwait(false) is no longer necessary in .NET Core. True?
False. It’s needed when running on .NET Core for exactly the same reasons it’s needed when running on .NET Framework. Nothing’s changed in that regard.

What has changed, however, is whether certain environments publish their own SynchronizationContext. In particular, whereas the classic ASP.NET on .NET Framework has its own SynchronizationContext, in contrast ASP.NET Core does not. That means that code running in an ASP.NET Core app by default won’t see a custom SynchronizationContext, which lessens the need for ConfigureAwait(false) running in such an environment.

It doesn’t mean, however, that there will never be a custom SynchronizationContext or TaskScheduler present. If some user code (or other library code your app is using) sets a custom context and calls your code, or invokes your code in a Task scheduled to a custom TaskScheduler, then even in ASP.NET Core your awaits may see a non-default context or scheduler that would lead you to want to use ConfigureAwait(false). Of course, in such situations, if you avoid synchronously blocking (which you should avoid doing in web apps regardless) and if you don’t mind the small performance overheads in such limited occurrences, you can probably get away without using ConfigureAwait(false).

@dtillman Can you chime in on this?

repo: SteeltoeOSS/Steeltoe


text:
Using NuGet Package with ClickOnce Deployment

Actually this seems not to be your fault, but it is not possible to use this nuget-package in combination with a clickonce deployment. Although there is no error, the problem is that ClickOnce does not see the dependency to MediaInfo.Native. Thus it does not add it to the installation and on application start all DLLs from MediaInfo.Native are not available and thus my application crashes.
I'm not an expert with NuGet, but do you think there is some other way to make your dependency on MediaInfo.Native more obvious? Or do you know of any way to properly install an application using your NuGet package?
I do need to find some way to install my application using your NuGet package and the required MediaInfo.dll (and its dependencies) in an easy way. - Thank you so much for your effort!

Quick question; did you try to build your app as x64 or 32 ?

Thanks for your immediate feedback. I tried both, but would be happy with x64 in the first try :-(

Am 27.03.2018 um 20:42 schrieb Stef Heyenrath notifications@github.com:
Quick question; did you try to build your app as x64 or 32 ?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Just created a quick example to show the problem: https://github.com/suchja/MediaInfoConsole
When preparing to deploy this solution the MediaInfo.Native package and its included DLLs like MediaInfo.dll do not appear in the corresponding wizard (sorry for the German UI).

I tried it once again with x86 and x64 and in addition with Debug and Release. It is the same in all combinations.
While inspecting the project file I found this entry. This seems to me like a first indication why all the other DLLs are not recognised by ClickOnce. You link the project which seems to be build during application build and not the already build DLL like you did it for MediaInfo.DotNetWrapper.dll.

A workaround which could work:
Just add the 5 dll's from the packages folder (C:\Users\azureuser\Documents\Github\MediaInfo.DotNetWrapper\packages\MediaInfo.Native.17.12\build\native\x86) or x64 to your project.
And set Copy if newer to true for all these dll's
When inspecting the clickone files, you see that the dll's are included:

Yeah, did that already as workaround. However, would be good to have a proper solution which builds for x86 and x64.

Another solution would be to add a post-build step and just copy the correct dll's from the packages\MediaInfo.Native.17.12\build\native... folder ?
Or conditionally include the dll's, by manually editing the csproj file?

That probably will do. Thank you! Will update this issue once I have it running, so the solution is documented.

authors: StefH suchja


text:
Question: Is it possible to call a generic method in a dynamic manner

Hello colleagues, sorry, the question maybe a bit silly, as i'm mot an experienced C# developer.
I got the following piece of code:
var result = list
 .Where(x => typesFiltered.Contains(x.Type))
 .Where(x => x.GetField<string>(ParameterName) == ParameterValue)
 .ToArray();

is it possible to convert it to something like
var result = list
 .Where(x => typesFiltered.Contains(x.Type))
 .Where("it.GetField<string>(ParameterName) == ParameterValue") // <- use the string here
 .ToArray();

If you just want to filter using this library, use code like:
var result = list
 .Where(x => typesFiltered.Contains(x.Type))
 .Where("it.ParameterName == \"ParameterValue\")
 .ToDynamicArray();

Closing...

authors: StefH eosfor


text:
Opbouw van de reservering's pagina met formulier

Hier kan de gebruiker zijn boek reserveren en post zijn gegevens in de api.

Deze zal Stefan op zich nemen.

authors: Stefan-Espant jtoufik

repo: Stefan-Espant/performance-matters-oba


text:
Use DeleteAclRequest_v1 if broker version is over 2.0.0

Fixes https://github.com/StephenSorriaux/ansible-kafka-admin/issues/61
Proposed Changes

Use DeleteAclRequest_v1 if broker version is over 2.0.0
(also forced EOL to LF, that's why the huge diff)

I tested this and I can confirm that it's now working with prefixed acls. Many thanks.

Thanks for your feedback, I will merge this and generate a new release in the next days

authors: StephenSorriaux justCatchingRye

repo: StephenSorriaux/ansible-kafka-admin


text:
crashing when custom font is set

maybe I'm doing this totally wrong. But I have a custom font and I need to set it. I tried passing just the string of the font. which i have in res/font/montessara_light_black as "montessara_light_black" in the setFont. It just crashed.
Can you show how to do that?

Indeed this supports fonts from asset folder. Using fonts from font folder was not possible when this library was created. This should be changed in order to support this

https://github.com/StephenVinouze/MaterialNumberPicker/pull/21

Cool! :D

thanks

authors: AlejandroHCruz StephenVinouze fperez-rsc githubashutoshsoni

repo: StephenVinouze/MaterialNumberPicker


text:
add checking for auth response

Signed-off-by: greg zimin_grigory@hotmail.com

good idea, but pls resolve conflicts

closed, decided not to merge it. Let's keep examples as minimal as possible.

authors: ZiminGrigory mehanig

repo: StepicOrg/Stepic-API


text:
🛑 DNS (he.net) is down

In b42247f, DNS (he.net) ($HE_NS) was down:

HTTP code: 0
Response time: 0 ms

Resolved: DNS (he.net) is back up in f4741f3.

repo: Sternwarte-St-Ottilien-e-V/status


text:
🛑 DNS (he.net) is down

In 19b6c0b, DNS (he.net) ($HE_NS) was down:

HTTP code: 0
Response time: 0 ms

Resolved: DNS (he.net) is back up in ba66094.


text:
🛑 DNS (he.net) is down

In d4560c8, DNS (he.net) ($HE_NS) was down:

HTTP code: 0
Response time: 0 ms

Resolved: DNS (he.net) is back up in 3364dd4.


text:
🛑 DNS (he.net) is down

In 3be5602, DNS (he.net) ($HE_NS) was down:

HTTP code: 0
Response time: 0 ms

Resolved: DNS (he.net) is back up in 1547aa7.


text:
🛑 DNS (he.net) is down

In 0767d24, DNS (he.net) ($HE_NS) was down:

HTTP code: 0
Response time: 0 ms

Resolved: DNS (he.net) is back up in 937dc0b.


text:
[歌词请愿] 阿云嘎, HOYO-MiX - Regression

歌曲名称
阿云嘎, HOYO-MiX - Regression
音乐对应的音乐平台和音乐 ID
1913478990
其他备注

[ ] 这个歌曲在 Apple Music 上有逐词歌词

备注
没有人做这首的话我就上传了哦😋

平台自带的我看着质量还行啊😂除了没有排好版之外😂

平台自带的我看着质量还行啊😂除了没有排好版之外😂

是的，因为排版所以我宁愿自己再手打一份（？）😂

是的，因为排版所以我宁愿自己再手打一份（？）😂

直接拿下来排一下版不就行了（

直接拿下来排一下版不就行了（

还要加一些背景人声吧，这两天看看啥情况吧🥲

#1857 Refered.

主教的爱，太沉重了...

repo: Steve-xmh/amll-ttml-db