text:
Sometimes incorrectly parsing output

I have some code that is currently calling getMAC() about once a second. Every once in a while it will return 'eth0' instead of the MAC address. It doesn't happen very often compared to how often it's called, but it happens a few times a day. This is running on a Debian Linux machine, btw.

We're hitting this problem too! No clue why it seems to be doing this. The first time i tired it returned the interface name every time, but on the second try it returns the correct mac address... If you have any insights into this let me know and i'll be happy to open a PR. We'll probably keep investigating.

Figured it out! PR incoming ;)

https://github.com/nekuz0r/node-arp/pull/8

authors: ZackMattor jwarkentin


text:
Suggestion: Rename of "Node" drop-down selection

Suggest to change "Node" to "Network Type/ Network Name" or "Generationhash" to reflect that those nodes could be from different network.
Maybe Network:Node?
e.g. Mainnet:http://13.114.200.132:3000/

hey there @ivyfung1 ; I would agree that there could be some information about the network name in the list.
But I think it is important to keep the node IPs there as this is the information that is relevant (which node is the explorer using to read the data I see.)

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
Harvesting confirmation view password input

Harvesting confirmation view password input is not visible and we need to scroll.
For other transaction types input is always visible.
Please watch video: https://share.getcloudapp.com/JruqbPel
Same with harvesting activation.

@NikolaiYurchenko could you check?

@NikolaiYurchenko I think we have still inconsistent/bad looking transaction confirmation pop-ups.

create aggregate pop-up -> https://share.getcloudapp.com/yAuZJbYn?utm_source=show
create aggregate (with multisig) -> https://share.getcloudapp.com/d5uPgp2L (three scrools)
how it looks on cosigner side (to sign) -> https://share.getcloudapp.com/p9urewoE (details cutted on top)
how it looks on cosigner side (which already signed tx) -> https://share.getcloudapp.com/NQuKW19m

Fixed.

repo: nemgrouplimited/symbol-desktop-wallet


text:
Mask not working as expected

I have the following json
{
  "tom": 3,
  "data1": {
    "first": "Hello",
    "second": {"third": "salude", "fourth": "nvm"}
   },
  "data2": {
    "first": "Bye",
    "second": {"third": "Cheers", "fourth": "nvm"}
   }
}

And I'm applying the follwing mask on it.
"data1(first,second/third),data2(first,second/third)"
I expected to see the,
{
  "data1": {
    "first": "Hello",
    "second": {"third": "salude"}
   },
  "data2": {
    "first": "Bye",
    "second": {"third": "Cheers"}
   }
}

but I'm only getting
{
  "data1": {
    "first": "Hello",
    "second": {"third": "salude"}
   }
}

Can you help me out?

@MathiasHaudgaard thanks for letting me know! That appears to be a bug. I can reproduce it.

Ok, please let me know when it's fixed. Thanks! :smiley:

@nemtsov It seems like the error comes from the buildTree functions. It exits the scope in the wrong order. When it comes to ")" it pops "/" and now it cant exit the last scope because of
if (peek && peek.tag === "/") doesn't match with "("

I'm bumping into the same error right now. Are there any updates here?

any updates?

Is the library abandonware?

Hey folks, although I don’t have a ton of time to work on this library myself, I’m more than happy to review PRs and help guide people interested in making contributions to it.

This the only bug i could find so far. Can you fix it?

What i found is, if / is not at the end of the query, what is next is going to be ignored.
Not working:
data1(first,second/third),data2(first,second/third)

Working:
data1(first,second),data2(first,second/third)

This is the only library that supports partial responses in plan JS. It is a shame just to abandon it.

This is resolved in v1.0.4 thanks for reporting it @MathiasHaudgaard

authors: MathiasHaudgaard alexrainman juan-restrepo mihanizm56 nemtsov

repo: nemtsov/json-mask

license: mit

license_source: bigquery


text:
CDC and Kafka Log Compaction

Hi everyone !! This issue is somehow similar to this one  #272 but the focus here is on Neo4j Streams Source for CDC in tandem with Kafka Log Compaction rather than on Neo4j Streams Procedures.
When using Change Data Capture with Kafka, one of the most common use cases is managing Kafka partitions as logs. In this scenario, Log Compaction is somehow necessary for CDC: we need a complete dataset but we must also avoid unlimited growth in the size of partitions. From the Kafka documentation:

Log compaction ensures that Kafka will always retain at least the last known value for each message key within the log of data for a single topic partition. It addresses use cases and scenarios such as restoring state after application crashes or system failure or reloading caches after application restarts during operational maintenance.

On data retention:

So far we have described only the simpler approach to data retention where old log data is discarded after a fixed period or when the log reaches some predetermined size. This works well for temporal event data such as logging where each record stands alone. However, an important class of data streams are the log of changes to keyed, mutable data (for example, the changes to a database table).

The current implementation of Neo4j Streams prevents this use cases since the message key is a combination of the transaction id and the id of the event inside the transaction, as shown in the following code (KafkaEventRouter.kt- line 86 - sendEvent  method):
private fun sendEvent(partition: Int, topic: String, event: StreamsTransactionEvent) {
        if (log.isDebugEnabled) {
            log.debug("Trying to send a transaction event with txId ${event.meta.txId} and txEventId ${event.meta.txEventId} to kafka")
        }
        val producerRecord = ProducerRecord(topic, partition, System.currentTimeMillis(),
                 "${event.meta.txId + event.meta.txEventId}-${event.meta.txEventId}",
                JSONUtils.writeValueAsBytes(event))
        send(producerRecord)
    }

As far as I know, this will generate a sequence of unique keys, preventing Log Compaction. Instead, choosing the node or relationship id as the message key ("${event.payload.id}") would enable Log Compaction on the Kafka side. However, I'm pretty sure there are specific reasons for creating the key that way: maybe the message keys are used to know where we are in the Neo4j log, to recover after a failure, but I'm not sure.
Can you explain why the keys are generated in this way? Is it possible to enable Log Compaction in Neo4j Streams right now? If the answer is no, I think this is a must-have feature for Change Data Capture. Thanks !!
Considered alternatives
While writing this issue, I actually came up with a possible alternative solution (the same also applies to the issue #304): one can simply create a microservice using Kafka Streams to add custom logic and cope with order guarantees and log compaction. However, I am still convinced that these problems should be addressed within Neo4j Streams.

@Solom00n thanks for the great analysis, we can do it via a configuration param in order to keep both options

@mroiter-larus is this issue close-able?

@moxious this had been put on hold that i know of

Left open pending duplicate functionality PR agaist 3.5 branch. Issue is closeable when 3.5 branch is merged with same fix/improvement.


text:
Can't run integration unittests

I have spend several hours trying to get the integration unittests to work. I'm running on a clean Windows 10 with Visual Studio 2015 Community and JRE 8. I'm running both VS, cosole and PS as administrator.
I have tracked down the issue to when the PS scripts tries to start the neo4j servies by debugging the unittest. Then I tried running the PS scripts manually and got the same error. Here is the log:

PS C:\Sources\neo4j\original\neo4j-dotnet-driver\Neo4j.Driver\Neo4j.Driver.IntegrationTests\bin\target\neo4j\neo4j-commu
nity-3.0.0-RC1\bin> Invoke-Neo4j start -v
VERBOSE: Neo4j Root is 'C:\Sources\neo4j\original\neo4j-dotnet-driver\Neo4j.Driver\Neo4j.Driver.IntegrationTests\bin\target\neo4j\neo4j-community-3.0.0-RC1'
VERBOSE: Neo4j Server Type is 'Community'
VERBOSE: Neo4j Version is '3.0.0-RC1'
VERBOSE: Neo4j Database Mode is ''
VERBOSE: Start command specified
VERBOSE: Neo4j Windows Service Name is neo4j
VERBOSE: Starting the service.  This can take some time...
Invoke-Neo4j : Failed to start service 'Neo4j Graph Database - neo4j (neo4j)'.
At line:1 char:1

Invoke-Neo4j start -v

  + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
  + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Neo4j

It works if I get it to run as console:

PS C:\Sources\neo4j\original\neo4j-dotnet-driver\Neo4j.Driver\Neo4j.Driver.IntegrationTests\bin\target\neo4j\neo4j-community-3.0.0-RC1\bin> Invoke-Neo4j console -v
VERBOSE: Neo4j Root is 'C:\Sources\neo4j\original\neo4j-dotnet-driver\Neo4j.Driver\Neo4j.Driver.IntegrationTests\bin\target\neo4j\neo4j-community-3.0.0-RC1'
VERBOSE: Neo4j Server Type is 'Community'
VERBOSE: Neo4j Version is '3.0.0-RC1'
VERBOSE: Neo4j Database Mode is ''
VERBOSE: Console command specified
VERBOSE: Java detected at 'C:\Program Files\Java\jre1.8.0_73\bin\java.exe'
VERBOSE: Java version detected as '1.8'
VERBOSE: Starting Neo4j as a console with command line C:\Program Files\Java\jre1.8.0_73\bin\java.exe -cp
"C:\Sources\neo4j\original\neo4j-dotnet-driver\Neo4j.Driver\Neo4j.Driver.IntegrationTests\bin\target\neo4j\neo4j-community-3.0.0-RC1/lib/;C:\Sources\neo4j\original\neo4j-dotnet-driver\Neo4j.D
river\Neo4j.Driver.IntegrationTests\bin\target\neo4j\neo4j-community-3.0.0-RC1/plugins/" -server -Dorg.neo4j.config.file=conf/neo4j.conf -Dlog4j.configuration=file:conf/log4j.properties
-Dneo4j.ext.udc.source=zip-powershell -Dorg.neo4j.cluster.logdirectory=data/log -Dorg.neo4j.config.file=conf/neo4j.conf -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:hashCode=5
-XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFinalNonStaticFields -XX:+DisableExplicitGC -Dunsupported.dbms.udc.source=zip -Dfile.encoding=UTF-8
org.neo4j.server.CommunityEntryPoint
2016-03-21 09:06:23.118+0000 INFO  Starting...
2016-03-21 09:06:31.680+0000 INFO  Started.
2016-03-21 09:06:34.633+0000 INFO  Remote interface available at http://localhost:7474/

I have also tried starting the service manually from Windows Services, but got this message:

The Neo4j Graph Database - neo4j service on Local Computer started and then stopped. Some services stop automatically if they are not in use by other services or programs.

I would really like to get them running, so I can contribute to the project :)
Any suggestions?

Hello, I looked into your problem yesterday and it seems you have some problem with Write-Error command used in our powershell script. The build also have some problem to use the recent powershell script so I am also wondering if I should just switch to use the bat file instead to avoid these changes to disturb the normal driver code. I am working on this problem and I will come back to you once I got a solution or work around. Will ping you when the build is ready for your PR.
Finally thanks a lot for the PRs.

Thanks for you feed back! Using the BAT file might fix the issues I have seen :)
As you might have seen by now, I have created a PR that does not use PS or the Windows Service for running the Neo4j server. I have tested this on 3 Window PCs and worked on them all - When using the original code, integration tests did not work on any of the 3 PCs. You can consider using this when not running on the build server (eg local testing) and use the original when running on the build server :)

https://github.com/neo4j/neo4j-dotnet-driver/pull/33

As the pr is already merged in, I will close this issue now.
Feel free to reopen it when you got problem again,

authors: IngvarKofoed zhenlineo


text:
Feature request: General async support

Since there is a release of the async bolt driver. It would be very useful to have it in OGM too.
https://neo4j.com/docs/developer-manual/current/drivers/sessions-transactions/#_asynchronous_programming

Thanks for your input.
Your options right now is using OGM through SDN and it's Spring based async support: Async query results.
We have also spiked some ideas around full reactive support: https://github.com/michael-simons/neo4j-reactive-java-client
I'm closing this issue for the time being. This doesn't mean we're not implementing async, we are tracking this already and having several such tickets around doesn't help. Thank you.

authors: michael-simons wem


text:
Applying each after variable length relationship ignores rel_length

When each is called after an association with variable length the rel_length parameter is ignored.
Additional information which could be helpful if relevant to your issue:
Code example (inline, gist, or repo)
[31] pry(main)> >> r.subordinates(rel_length: { min: 0 }).each.count
 Role#subordinates
  MATCH (previous:`Role`)
  WHERE (ID(previous) = {ID_previous})
  OPTIONAL MATCH (previous)<-[rel1:`manager`]-(next:`Role`)
  RETURN
    ID(previous),
    collect(next) | {:ID_previous=>14930}
 HTTP REQUEST: 4ms POST http://localhost:7474/db/data/transaction/commit (1 bytes)
=> 6
[32] pry(main)> >> r.subordinates(rel_length: { min: 0 }).count
 Role#subordinates
  MATCH (role14930)
  WHERE (ID(role14930) = {ID_role14930})
  MATCH (role14930)<-[rel1:`manager`*0..]-(result_subordinates:`Role`)
  RETURN count(result_subordinates) AS result_subordinates | {:ID_role14930=>14930}
 HTTP REQUEST: 7ms POST http://localhost:7474/db/data/transaction/commit (1 bytes)
=> 635

Runtime information:
Neo4j database version:
neo4j gem version: 8.0.17
neo4j-core gem version: 7.1.2

For reference a failing spec:
describe 'bug' do
  before(:each) do
    clear_model_memory_caches
    delete_db

    stub_active_node_class('Person') do
      has_many :out, :knows, model_class: 'Person', type: nil
    end

    stub_active_node_class('Company') do
      has_one :out, :ceo, type: :ceo, model_class: 'Person'
    end
  end

  let(:ceo) { Person.create }
  let(:company) { Company.create(ceo: ceo) }

    it 'should not drop rel_length' do
      expect(ceo.knows(rel_length: { min: 0 }).count).to eq(1)
      expect(ceo.knows(rel_length: { min: 0 }).to_a.count).to eq(1)

      ceo_via_association = company.ceo
      expect(ceo_via_association).to eq(ceo)

      expect(ceo_via_association.knows(rel_length: { min: 0 }).count).to eq(1)
      expect(ceo_via_association.knows(rel_length: { min: 0 }).to_a.count).to eq(1) # fails returning 0 as the automated 1 + n prevention kicks in and ignores the parameters of knows
    end
end


text:
Video Library Documentation

The Documentation must contain:

Detailed API Documentation
Examples of API usage (Auth, etc. concepts)
How to add custom Database
How to alter the Mock Backend

Develop using Docusaurus 2.0

Fixed #14

repo: neogcamp/mockBee

license: MIT


text:
Preserve line-breaks in error messages

Neomake loses line breaks when displaying errors. Using Haskell as an example:
Neomake w/ hdevtools:

Neomake w/ ghcmod:

Expected behavior
ghcmod-vim preserves line breaks when displaying errors:

History
This seems to be a well-known problem.
When Haskell Makers were first being added to Neomake, there was no attempt to preserve line breaks. However, there were problems capturing every line of a multi-line error, much like Syntastic experienced. @benekastah referenced #15, which was eventually resolved, but it appears the accepted solution was to join lines into one.
In the ghcmod-vim project, @toonn expressed a preference for using that plugin with Syntastic for a few reasons which included preserving line breaks. @expipiplus1 suggested Neomake, but admitted it "still munges everything into a single line". He cited a Syntastic issue on the subject of preserving line breaks, in which @lcd047 imparted some crucial wisdom:

What ghcmod-vim shows are valid error lines intermixed with invalid ones. You'll notice that going to, say, line 2 in the quickfix list and pressing Enter doesn't take you to the error line in the source. Whether this is better than syntastic's UI or not is up to debate.

@expipiplus1 offered an alternative:

ghcmod-vim could work around this by making each individual line from an error it's own proper error message

The downside of this approach is that :lnext won't jump to the next error until after you get through all the lines of a multi-line message.
The code to get ghcmod-vim's behavior exists, but it's a bit gnarly. Messages are first saved to a file, then read with readfile (which corrects null characters to newlines), and then lines after the first are added to the quickfix list without file or line number.
These seem to be two non-ideal alternatives for multi-line messages (to use "invalid" lines or not), each with their own drawbacks but both, in my opinion, preferable to the status quo. A capable framework could provide all three as options to the user. If I wanted to pursue that route, where would I find the best spot in the code to hook a single-message-to-possibly-singleton-list-of-quickfix-items function?
Steps to reproduce
" Use one of the below settings and :Neomake.
let g:neomake_haskell_enabled_makers = ['hdevtools']
let g:neomake_haskell_enabled_makers = ['ghcmod']

Output of the ":verb NeomakeInfo" command
Neomake debug information
Async support: 0
Current filetype:
Enabled makers
For the current filetype (with :Neomake): []
NOTE: the current buffer does not have a filetype.
For the project (with :Neomake!): []
NOTE: you can define g:neomake_enabled_makers to configure it.
Settings
g:neomake_haskell_enabled_makers = ['ghcmod']

shell: /bin/bash
shellcmdflag: -c
Windows: 0

:version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Nov 24 2016 16:44:48)
Included patches: 1-1689
Extra patches: 8.0.0056
Modified by pkg-vim-maintainers@lists.alioth.debian.org
Compiled by pkg-vim-maintainers@lists.alioth.debian.org
Huge version with GTK2-GNOME GUI.  Features included (+) or not (-):
+acl             +conceal         +file_in_path    +linebreak       -mouse_sysmouse  +python3         +tcl             +wildmenu
+arabic          +cryptv          +find_in_path    +lispindent      +mouse_urxvt     +quickfix        +terminfo        +windows
+autocmd         +cscope          +float           +listcmds        +mouse_xterm     +reltime         +termresponse    +writebackup
+balloon_eval    +cursorbind      +folding         +localmap        +multi_byte      +rightleft       +textobjects     +X11
+browse          +cursorshape     -footer          +lua             +multi_lang      +ruby            +timers          -xfontset
++builtin_terms  +dialog_con_gui  +fork()          +menu            -mzscheme        +scrollbind      +title           +xim
+byte_offset     +diff            +gettext         +mksession       +netbeans_intg   +signs           +toolbar         +xsmp_interact
+channel         +digraphs        -hangul_input    +modify_fname    +packages        +smartindent     +user_commands   +xterm_clipboard
+cindent         +dnd             +iconv           +mouse           +path_extra      +startuptime     +vertsplit       -xterm_save
+clientserver    -ebcdic          +insert_expand   +mouseshape      +perl            +statusline      +virtualedit     +xpm
+clipboard       +emacs_tags      +job             +mouse_dec       +persistent_undo -sun_workshop    +visual
+cmdline_compl   +eval            +jumplist        +mouse_gpm       +postscript      +syntax          +visualextra
+cmdline_hist    +ex_extra        +keymap          -mouse_jsbterm   +printer         +tag_binary      +viminfo
+cmdline_info    +extra_search    +langmap         +mouse_netterm   +profile         +tag_old_static  +vreplace
+comments        +farsi           +libcall         +mouse_sgr       -python          -tag_any_white   +wildignore
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  system gvimrc file: "$VIM/gvimrc"
    user gvimrc file: "$HOME/.gvimrc"
2nd user gvimrc file: "~/.vim/gvimrc"
    system menu file: "$VIMRUNTIME/menu.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -DFEAT_GUI_GTK  -pthread -I/usr/include/gtk-2.0 -I/usr/lib/x86_64-linux-gnu/gtk-2.0/inclu
de -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/
pixman-1 -I/usr/include/libpng12 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng12 -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -
I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/freetype2 -D_REENTRANT -DORBI
T2=1 -pthread -I/usr/include/libgnomeui-2.0 -I/usr/include/gnome-keyring-1 -I/usr/include/libbonoboui-2.0 -I/usr/include/libxml2 -I/usr/i
nclude/libgnome-2.0 -I/usr/include/libbonobo-2.0 -I/usr/include/bonobo-activation-2.0 -I/usr/include/orbit-2.0 -I/usr/include/libgnomecan
vas-2.0 -I/usr/include/gail-1.0 -I/usr/include/libart-2.0 -I/usr/include/gtk-2.0 -I/usr/lib/x86_64-linux-gnu/gtk-2.0/include -I/usr/inclu
de/gio-unix-2.0/ -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr
/include/libpng12 -I/usr/include/pango-1.0 -I/usr/include/harfbuzz -I/usr/include/pango-1.0 -I/usr/include/freetype2 -I/usr/include/gdk-p
ixbuf-2.0 -I/usr/include/libpng12 -I/usr/include/gnome-vfs-2.0 -I/usr/lib/x86_64-linux-gnu/gnome-vfs-2.0/include -I/usr/include/gconf/2 -
I/usr/include/dbus-1.0 -I/usr/lib/x86_64-linux-gnu/dbus-1.0/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include
-Wdate-time  -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc   -L. -Wl,-Bsymbolic-functions -Wl,-z,relro -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,-E  -Wl,-Bsymbolic-functions
 -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -o vim   -lgtk-x11-2.0 -lgdk-x11-2.0 -lpangocairo-1.0 -latk-1.0 -lcairo -lgdk_pixbuf-
2.0 -lgio-2.0 -lpangoft2-1.0 -lpango-1.0 -lgobject-2.0 -lglib-2.0 -lfontconfig -lfreetype  -lgnomeui-2 -lSM -lICE -lbonoboui-2 -lgnome-2
-lpopt -lbonobo-2 -lbonobo-activation -lORBit-2 -lgnomecanvas-2 -lart_lgpl_2 -lgtk-x11-2.0 -lgdk-x11-2.0 -lpangocairo-1.0 -latk-1.0 -lcai
ro -lgio-2.0 -lpangoft2-1.0 -lpango-1.0 -lfontconfig -lfreetype -lgdk_pixbuf-2.0 -lgnomevfs-2 -lgconf-2 -lgthread-2.0 -lgmodule-2.0 -lgob
ject-2.0 -lglib-2.0 -lSM -lICE -lXpm -lXt -lX11 -lXdmcp -lSM -lICE  -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl  -L/usr/lib -ll
ua5.2 -Wl,-E  -fstack-protector-strong -L/usr/local/lib  -L/usr/lib/x86_64-linux-gnu/perl/5.22/CORE -lperl -ldl -lm -lpthread -lcrypt  -L
/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu -lpython3.5m -lpthread -ldl -lutil -lm -L/usr/lib/x86_64-linux-gnu -ltcl8.6 -ldl -lz -lpt
hread -lieee -lm -lruby-2.3 -lpthread -lgmp -ldl -lcrypt -lm

I wonder if it'd be possible to replace lnext with a version which skips to the next qf line with a location.

One potential solution here is to extend neomake with a new command that can pretty-print the current error. This seems like a good way to work around the fact that vim quickfix can only store a single line.
Another idea could be to propose this as an enhancement to neovim - perhaps they would be willing to move quickfix in this direction.

authors: dlants expipiplus1 thejohnfreeman

repo: neomake/neomake


text:
pageserver: clean up ancestral layers after split, old index_part objects

Per the "Cleaning up parent-shard layers" section in #6358 -- currently after a shard split, layers from the parent shards are not deleted until the whole tenant is eventually deleted.
We should implement an occasional online scrub routine that checks which of these are referenced by children, and cleans them up.
It likely makes sense to combine this work with cleaning up old-generation index_part.json objects, as these older objects will likely reference parent shard layers -- we should first define the criteria for cleaning up old indices, and then use the still-alive indices as the source of references for cleaning up parent layers.
### Tasks
- [ ] https://github.com/neondatabase/neon/pull/7925
- [ ] https://github.com/neondatabase/neon/pull/8196
- [ ] https://github.com/neondatabase/cloud/issues/14024

This will be enabled in staging here: https://github.com/neondatabase/aws/pull/1654
Then we'll let it soak for at least a week before proceeding to prod.

repo: neondatabase/neon


text:
Replace 'tar' crate with 'tokio-tar'

The synchronous 'tar' crate has required us to use block_in_place and SyncIoBridge to work together with the async I/O in the client connection. Switch to 'async-tar' crate that uses async I/O natively.
As part of this, move the CopyDataWriter implementation to postgres_backend_async.rs. Even though it's only used in one place currently, it's in principle generally applicable whenever you want to use COPY out.

'async-tar' is a fork of the 'tar' crate, just replacing sync functions with corresponding async ones. I'm not sure how well maintained it is, but a crate like this doesn't really need much change or maintenance. If we go with this, it might be good to take a close look at 'async-tar' and compare if it's missing any fixes that have been made on 'tar' crate.
I'm surprised by how many dependencies the 'async-tar' crate pulls in. It depends on 'async-std'; is that really necessary, or could we update 'async-tar' to rely on the corresponding std functionality instead?

It depends on 'async-std'

Let's not pull that in then, it's yet another async runtime and we have tokio for that.
Could something like https://crates.io/crates/tokio-tar be a better choice?

It depends on 'async-std'

Let's not pull that in then, it's yet another async runtime and we have tokio for that.

I see.

Could something like https://crates.io/crates/tokio-tar be a better choice?

Tried that now. There's one problem with it:
error[E0477]: the type `AbortableWrite<'a, W>` does not fulfill the required lifetime
  --> pageserver/src/basebackup.rs:44:9
   |
44 |     ar: Builder<AbortableWrite<'a, W>>,
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: type must satisfy the static lifetime as required by this binding
  --> /home/heikki/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-tar-0.3.0/src/builder.rs:15:46
   |
15 | pub struct Builder<W: Write + Unpin + Send + 'static> {
   |                                              ^^^^^^^

tokio_tar::Builder requires the writer to be 'static. The reason for that is that it has a mechanism to write the tar EOF block (1024 bytes of zeros), if you just drop the Builder without calling finish. The plain tar crate has that too, and it's straightforward there: the Drop implementation writes the EOF marker. In the async_tar, the Drop implementation uses block_on to do the same. But in tokio_tar, it uses a channel and a separately spawned task to do it.
That's hacky IMHO. The EOF-block-writing-task executes at some not-well defined time after the drop has happened. And it's super annoying for us, because we actually don't want the EOF marker to be written at all. We actually work hard to skip it, that's exactly why we have the AbortableWrite hack.
So ideally tokio-tar just didn't have that Drop implementation, and then it wouldn't require 'static, and we could also remove AbortableWrite. I'm tempted to fork it and do that. Perhaps the upstream project would be interested in a PR for that too; IMHO it's a bad idea to write the EOF block on drop anyway.

Switched to a modified version of 'tokio-tar' without the Drop implementation and the 'static requirement.

Looking more, https://github.com/vorot93/tokio-tar/pull/9 does look quite important in case the work not completing on one poll, however that's only on the read path which we don't use.
Looking at tokio-tar the AsyncWrite usage of slices is not cancellation safe. I doubt know if it was cancellation safe before, but now it definitely is not. I think however the only place it's cancelled is with the shutdown waiter, and then the writer is not restarted back again. Maybe a comment or two could be added about this near... I'll try to find that callsite.

We don't use that Drop path indeed, and I've created a PR that should explicitly help us with that: https://github.com/neondatabase/tokio-tar/pull/1 , there are more details about the upstreaming activities, let's see if they ever get released.
For now, out fork can do that: https://github.com/neondatabase/neon/pull/3239

source: gharchive/pull-request

authors: SomeoneToIgnore hlinnaka koivunej


text:
fix: connection string uri scheme to postgresql

Use postgresql scheme for connection string as postgres causes issue with some tools
Validate database name passed to connection-string command

:tada: This PR is included in version 1.29.4 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

authors: Shridhad duskpoet

repo: neondatabase/neonctl


text:
Fatal error: Class 'TYPO3\SwiftMailer\Message' not found

This line in the EmailFinisher class does not ensure that the required Swiftmailer class is actually available (and causes a fatal error if you don't have Swiftmailer installed):
$mail = new \TYPO3\SwiftMailer\Message();
I suggest to either include Swiftmailer as dependency in composer.json or to check for installed Swiftmailer and throw an informative exception instead.

Thanks, good point! I guess we should at least suggest the switftmailer package if we don't want to create a hard dependency.

Fixed with #23

authors: KatharinaSt kdambekalns kitsunet

repo: neos/form


text:
BUGFIX: Prevent Fatal Error in Setup

This is a hotfix for a regression introduced with https://github.com/neos/flow-development-collection/pull/499.
Background:
The latest Http\RequestHandler of Flow internally uses
the ComponentChain to determine the current Request/Response.
This package uses a specialized handler which doesn't rely on
the ComponentChain leading to a Fatal Error.

needs to go into the 2.0 branch right?

needs to go into the 2.0 branch right?

I'm not  sure, 2.0 is used in Neos 2.0+ and the fix is only required for Neos 2.3.x. I guess we should probably create a 2.3 branch in the setup and apply it there!?

The evils of inheritance... Makes sense, but be aware that the $request and $response properties will also be removed with 4.0 - so maybe already copy them over

be aware that the $request and $response properties will also be removed with 4.0

That's news to me. Is that documented somewhere or is there a PR for this?
Anyways, I rather merge this now to fix the fatal.
I'd suggest to merge it into master and then create a branch 2.3 and require that in the affected distributions. Does that make sense?

Is that documented somewhere or is there a PR for this?

https://github.com/neos/flow-development-collection/pull/500

@bwaidelich: I'd rather avoid a separate version if it can be done in backwards compatible way, which this seems to be..

I messed up the rebase, see #28

authors: aertmann albe bwaidelich

repo: neos/setup


text:
:LspRestart throws error when using non-supported language server

Description
When called without arguments in a buffer attached to a non-supported language server, :LspRestart throws the following error:
Error executing vim.schedule lua callback: .../site/pack/packer/start/nvim-lspconfig/lua/lspconfig.lua:58: attempt to index a nil value
stack traceback:
        .../site/pack/packer/start/nvim-lspconfig/lua/lspconfig.lua:58: in function ''
        vim.lua: in function ''
        vim.lua: in function <vim.lua:0>

Neovim version
NVIM v0.7.0-dev+732-g80210c189
Nvim-lspconfig version
64bd3b95345f2dbedc83a3d9a472662c48b3e940
Operating system and version
macOS 12.1
Affected language servers
All
Steps to reproduce

nvim --clean -u minimal_init.lua test.py
:LspRestart

Actual behavior
The error from the description is thrown.
Expected behavior
No error is thrown.
Minimal config
local on_windows = vim.loop.os_uname().version:match("Windows")

local function join_paths(...)
    local path_sep = on_windows and "\\" or "/"
    local result = table.concat({ ... }, path_sep)
    return result
end

vim.cmd([[set runtimepath=$VIMRUNTIME]])

local temp_dir
if on_windows then
    temp_dir = vim.loop.os_getenv("TEMP")
else
    temp_dir = "/tmp"
end

vim.cmd("set packpath=" .. join_paths(temp_dir, "nvim", "site"))

local package_root = join_paths(temp_dir, "nvim", "site", "pack")
local install_path = join_paths(package_root, "packer", "start", "packer.nvim")
local compile_path = join_paths(install_path, "plugin", "packer_compiled.lua")

local function load_plugins()
    require("packer").startup({
        {
            "wbthomason/packer.nvim",
            "neovim/nvim-lspconfig",
        },
        config = {
            package_root = package_root,
            compile_path = compile_path,
        },
    })
end

_G.attach_buffer = function()
    local client_id = vim.lsp.start_client({ cmd = { "pylsp" } }) -- cmd is irrelevant
    vim.lsp.buf_attach_client(vim.api.nvim_get_current_buf(), client_id)
end

_G.load_config = function()
    -- filetype is irrelevant but should be consistent with language server
    vim.cmd("autocmd FileType python lua attach_buffer()")
end

if vim.fn.isdirectory(install_path) == 0 then
    vim.fn.system({ "git", "clone", "https://github.com/wbthomason/packer.nvim", install_path })
    load_plugins()
    require("packer").sync()
    vim.cmd([[autocmd User PackerComplete ++once lua load_config()]])
else
    load_plugins()
    require("packer").sync()
    _G.load_config()
end

Health check
lspconfig: require("lspconfig.health").check()
========================================================================
  - WARNING: Can't find any config.

LSP log
Probably not relevant

The following trivial patch fixes the issue as well as a related issue where a config is defined but setup has not yet been called, meaning that launch is not yet defined. I didn't yet submit a PR in case I'm missing something but am happy to do so if this is the right fix.
diff --git a/lua/lspconfig.lua b/lua/lspconfig.lua
index 8404632..685b166 100644
--- a/lua/lspconfig.lua
+++ b/lua/lspconfig.lua
@@ -54,9 +54,11 @@ function M._root._setup()
       function(cmd_args)
         for _, client in ipairs(M.util.get_clients_from_cmd_args(cmd_args)) do
           client.stop()
-          vim.defer_fn(function()
-            configs[client.name].launch()
-          end, 500)
+          if configs[client.name] and configs[client.name].launch then
+            vim.defer_fn(function()
+              configs[client.name].launch()
+            end, 500)
+          end
         end
       end,
       '-nargs=? -complete=customlist,v:lua.lsp_get_active_client_ids',

I think a cleaner "fix" would be to expose the table of client ids managed by lspconfig, and make these methods only restart those clients.

Sure, that makes sense. Does this table already exist somewhere in the code, or is it something that would need to be created?

I think a cleaner "fix" would be to expose the table of client ids managed by lspconfig, and make these methods only restart those clients.

How about using the docs tag? or maybe add a special marker in the client config. Otherwise, how would you know which ones are managed? just because tsserver is active, doesn't mean it's managed by lspconfig.

Because lspconfig tracks internally which clients it starts... Please read the code and evaluate the solution I suggested before theorycrafting alternatives.

I took another look, and I can see that the active client IDs are available via manager.clients(). Does something like this make sense?
local get_lspconfig_clients = function()
    local clients = {}
    for _, config in pairs(require("lspconfig.configs")) do
        if config.manager then
            vim.list_extend(clients, config.manager.clients())
        end
    end
    return clients
end

If so, using that here instead of vim.lsp.get_active_clients() would solve this specific issue.

Yup

authors: jose-elias-alvarez kylo252 mjlbach

repo: neovim/nvim-lspconfig


text:
Error when decoding stdout from rust-analyzer

Description
Every time when I open a .rs file, the following warning occurs:
[lspconfig] unhandled error: ...ig/lua/lspconfig/server_configurations/rust_analyzer.lua:41: Expected value but found T_END at character 1

and rust-analyzer fails to launch.
The cause of the problem lies in lua/lspconfig/server_configurations/rust_analyzer.lua line 41 where we try to call vim.json.decode with parameter "" (empty string). How can we solve this?
Neovim version
NVIM v0.10.0-dev-311+g03ca36d1f8
Build type: Release
LuaJIT 2.1.0-beta3
Nvim-lspconfig version
84717430fb6f51c9e9cc8d63da10cceafc91500c
Operating system and version
Linux 6.3.1-arch1-1
Affected language servers
rust-analyzer
Steps to reproduce

nvim -nu minimal.lua
Open an arbitrary .rs file
Observe the warning

Actual behavior
rust-analyzer fails to launch.
Expected behavior
rust-analyzer launches and attaches successfully.
Minimal config
---@diagnostic disable: undefined-global
local on_windows = vim.loop.os_uname().version:match 'Windows'

local function join_paths(...)
  local path_sep = on_windows and '\\' or '/'
  local result = table.concat({ ... }, path_sep)
  return result
end

vim.cmd [[set runtimepath=$VIMRUNTIME]]

local temp_dir = vim.loop.os_getenv 'TEMP' or '/tmp'

vim.cmd('set packpath=' .. join_paths(temp_dir, 'nvim', 'site'))

local package_root = join_paths(temp_dir, 'nvim', 'site', 'pack')
local lspconfig_path = join_paths(package_root, 'test', 'start', 'nvim-lspconfig')

if vim.fn.isdirectory(lspconfig_path) ~= 1 then
  vim.fn.system { 'git', 'clone', 'https://github.com/neovim/nvim-lspconfig', lspconfig_path }
end

vim.lsp.set_log_level 'trace'
require('vim.lsp.log').set_format_func(vim.inspect)
local nvim_lsp = require 'lspconfig'
local on_attach = function(_, bufnr)
  local function buf_set_option(...)
    vim.api.nvim_buf_set_option(bufnr, ...)
  end

  buf_set_option('omnifunc', 'v:lua.vim.lsp.omnifunc')

  -- Mappings.
  local opts = { buffer = bufnr, noremap = true, silent = true }
  vim.keymap.set('n', 'gD', vim.lsp.buf.declaration, opts)
  vim.keymap.set('n', 'gd', vim.lsp.buf.definition, opts)
  vim.keymap.set('n', 'K', vim.lsp.buf.hover, opts)
  vim.keymap.set('n', 'gi', vim.lsp.buf.implementation, opts)
  vim.keymap.set('n', '<C-k>', vim.lsp.buf.signature_help, opts)
  vim.keymap.set('n', '<space>wa', vim.lsp.buf.add_workspace_folder, opts)
  vim.keymap.set('n', '<space>wr', vim.lsp.buf.remove_workspace_folder, opts)
  vim.keymap.set('n', '<space>wl', function()
    print(vim.inspect(vim.lsp.buf.list_workspace_folders()))
  end, opts)
  vim.keymap.set('n', '<space>D', vim.lsp.buf.type_definition, opts)
  vim.keymap.set('n', '<space>rn', vim.lsp.buf.rename, opts)
  vim.keymap.set('n', 'gr', vim.lsp.buf.references, opts)
  vim.keymap.set('n', '<space>e', vim.diagnostic.open_float, opts)
  vim.keymap.set('n', '[d', vim.diagnostic.goto_prev, opts)
  vim.keymap.set('n', ']d', vim.diagnostic.goto_next, opts)
  vim.keymap.set('n', '<space>q', vim.diagnostic.setloclist, opts)
end

-- Add the server that troubles you here
local name = 'rust_analyzer'
local cmd = { 'rust_analyzer' } -- needed for elixirls, lua_ls, omnisharp
if not name then
  print 'You have not defined a server name, please edit minimal_init.lua'
end
if not nvim_lsp[name].document_config.default_config.cmd and not cmd then
  print [[You have not defined a server default cmd for a server
    that requires it please edit minimal_init.lua]]
end

nvim_lsp[name].setup {
  cmd = cmd,
  on_attach = on_attach,
}

print [[You can find your log at $HOME/.cache/nvim/lsp.log. Please paste in a github issue under a details tag as described in the issue template.]]

LSP log
https://gist.github.com/Bekaboo/f727c89767529c9f532c723e6f1a06be

Additional info: this warning only shows when opening bare .rs files (i.e. not in a rust project).


text:
WebSocket support

Create interfaces for client/Client and neptulon/Listener objects and separate implementation as:

client/TCPClient & client/WSClient
neptulon/TCPListener & neptulon/WSListener

We can even skip this step and keep everything all in one if things are as easy as what we did with ListenTLS.

We could also change project description after this to:

TCP/WebSocket framework for Go.
Minimalist web framework for Go.

Done.

repo: neptulon/neptulon


authors: advanderveer neoel

repo: nerdalize/nerd


text:
Fix on China AWS support

if run Yace on china AWS, there is error “Couldn't describe resources for region cn-northwest-1: InvalidClientTokenId: The security token included in the request is invalid.\n\tstatus code: 403,” and china AWS is in seperate infra.
Session new need to specified region, we fixed code based on Release 0.26.3-alpha, it has been tested successfully in China.
and in aws_cloudwatch.go ln50:
func createStsSession(roleArn string)
this part of code seems new adding, and region is unnecessary here. It has been tested.

Was fixed and merged

the same issue ; {"arn":"","error":"InvalidClientTokenId: The security token included in the request is invalid.\n\tstatus code: 403, request id: 46cbd89f-1a5d-49b4-996f-c2b616f4479c","job_type":"alb","level":"error","msg":"Couldn't get account Id","region":"cn-northwest-1","time":"2023-10-24T11:43:05Z"}.

authors: alibner insectme thomaspeitz


text:
OSError: Unknown compute capability. Ensure PyTorch with CUDA support is installed.

Installed nerfstudio with CUDA 11.3 as per setup page. Everything works fine. Followed the tutorial all the way to ns-train nerfacto --data data/nerfstudio/poster where I get this error:
$ ns-train nerfacto --data data/nerfstudio/poster
Traceback (most recent call last):
  File "C:\Users\glens\miniconda3\envs\ns\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\glens\miniconda3\envs\ns\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\glens\miniconda3\envs\ns\Scripts\ns-train.exe\__main__.py", line 4, in <module>
  File "C:\Users\glens\miniconda3\envs\ns\lib\site-packages\scripts\train.py", line 50, in <module>
    from nerfstudio.configs.method_configs import AnnotatedBaseConfigUnion
  File "C:\Users\glens\miniconda3\envs\ns\lib\site-packages\nerfstudio\configs\method_configs.py", line 46, in <module>
    from nerfstudio.field_components.temporal_distortions import TemporalDistortionKind
  File "C:\Users\glens\miniconda3\envs\ns\lib\site-packages\nerfstudio\field_components\__init__.py", line 17, in <module>
    from .encodings import Encoding, ScalingAndOffset
  File "C:\Users\glens\miniconda3\envs\ns\lib\site-packages\nerfstudio\field_components\encodings.py", line 34, in <module>
    import tinycudann as tcnn
  File "C:\Users\glens\miniconda3\envs\ns\lib\site-packages\tinycudann\__init__.py", line 9, in <module>
    from tinycudann.modules import free_temporary_memory, NetworkWithInputEncoding, Network, Encoding
  File "C:\Users\glens\miniconda3\envs\ns\lib\site-packages\tinycudann\modules.py", line 16, in <module>
    raise EnvironmentError("Unknown compute capability. Ensure PyTorch with CUDA support is installed.")
OSError: Unknown compute capability. Ensure PyTorch with CUDA support is installed.

Removed CUDA 11.3, installed 11.7 and followed the install instructions on the Pytorch page, i.e. conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia. Everything works as per the pip install instructions. tinycudann even seemed to build a lot faster. Ended up with exactly the same error: OSError: Unknown compute capability. Ensure PyTorch with CUDA support is installed.
CUDA variables are set as (not sure if this is relevant):

I understand devs are busy and I don't want to bother anyone but I'm trying really hard to get this working and just keep getting pipped at the post. Let me know if there's a better forum.

I ran into the same exact error today with conda installation of PyTorch with CUDA 11.7 (11.3 worked fine for me though). Seems like a problem with tinycuda rather than nerfstudio.

I'm also using CUDA 11.3, did the tinycuda installation go entirely smoothly?

@THE-COB I have both had tinycudann installation go smoothly and not work at all. I think @brawlawnzo is right, it's an issue with tinycuda.
I just tried to install as per nerfstudio installation instructions on my work computer. Visual Studio 2019, CUDA 11.3, Python 3.8. Completely clean install. Got this error install tinycudann:
 pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
Collecting git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
  Cloning https://github.com/NVlabs/tiny-cuda-nn/ to c:\users\glens\appdata\local\temp\pip-req-build-s1rw0r85
  Running command git clone --filter=blob:none --quiet https://github.com/NVlabs/tiny-cuda-nn/ 'C:\Users\glens\AppData\Local\Temp\pip-req-build-s1rw0r85'
  Resolved https://github.com/NVlabs/tiny-cuda-nn/ to commit 14053e9a87ebf449d32bda335c0363dd4f5667a4
  Running command git submodule update --init --recursive -q
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: tinycudann
  Building wheel for tinycudann (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [1327 lines of output]
      Building PyTorch extension for tiny-cuda-nn version 1.7
      Obtained compute capability 86 from PyTorch
      running bdist_wheel
      C:\Users\glens\miniconda3\envs\ns\lib\site-packages\torch\utils\cpp_extension.py:411: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
        warnings.warn(msg.format('we could not find ninja.'))
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-cpython-38
      creating build\lib.win-amd64-cpython-38\tinycudann
      copying tinycudann\modules.py -> build\lib.win-amd64-cpython-38\tinycudann
      copying tinycudann\__init__.py -> build\lib.win-amd64-cpython-38\tinycudann
      running egg_info
      creating tinycudann.egg-info
      writing tinycudann.egg-info\PKG-INFO
      writing dependency_links to tinycudann.egg-info\dependency_links.txt
      writing top-level names to tinycudann.egg-info\top_level.txt
      writing manifest file 'tinycudann.egg-info\SOURCES.txt'
      reading manifest file 'tinycudann.egg-info\SOURCES.txt'
      writing manifest file 'tinycudann.egg-info\SOURCES.txt'
      copying tinycudann\bindings.cpp -> build\lib.win-amd64-cpython-38\tinycudann
      running build_ext
      building 'tinycudann_bindings_86._C' extension
      creating build\dependencies
      creating build\dependencies\fmt
      creating build\dependencies\fmt\src
      creating build\src
      creating build\temp.win-amd64-cpython-38
      creating build\temp.win-amd64-cpython-38\Release
      creating build\temp.win-amd64-cpython-38\Release\tinycudann
      "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\gle...

followed by thousands of lines of errors, and then:
      Error limit reached.
      100 errors detected in the compilation of "../../src/common.cu".
      Compilation terminated.
      common.cu
      error: command 'C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA\\v11.3\\bin\\nvcc.exe' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> tinycudann

@grshennan
I think I might have run into this as well!
It seems to resemble the issue in here: https://github.com/NVlabs/tiny-cuda-nn/issues/169
One of the things in that thread that I think got me passed that (and then to the next error lol) was conda installing ninja.
If that doesn't work maybe try reinstalling visual studio (2019 should be alright, but I'm currently using 2017)

@THE-COB I have VS 2019 installed but same idea, yes. I think it's the same as this issue: https://github.com/NVlabs/tiny-cuda-nn/issues/147
Tried installing ninja and got a different error. Now it's
running build_ext
error: [WinError 2] The system cannot find the file specified

as per the tinycudann issue.

What is your current error? @grshennan

$ pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
Collecting git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
  Cloning https://github.com/NVlabs/tiny-cuda-nn/ to c:\users\glens\appdata\local\temp\pip-req-build-6w2nnebl
  Running command git clone --filter=blob:none --quiet https://github.com/NVlabs/tiny-cuda-nn/ 'C:\Users\glens\AppData\Local\Temp\pip-req-build-6w2nnebl'
  Resolved https://github.com/NVlabs/tiny-cuda-nn/ to commit 14053e9a87ebf449d32bda335c0363dd4f5667a4
  Running command git submodule update --init --recursive -q
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: tinycudann
  Building wheel for tinycudann (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      Building PyTorch extension for tiny-cuda-nn version 1.7
      Obtained compute capability 86 from PyTorch
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-cpython-38
      creating build\lib.win-amd64-cpython-38\tinycudann
      copying tinycudann\modules.py -> build\lib.win-amd64-cpython-38\tinycudann
      copying tinycudann\__init__.py -> build\lib.win-amd64-cpython-38\tinycudann
      running egg_info
      creating tinycudann.egg-info
      writing tinycudann.egg-info\PKG-INFO
      writing dependency_links to tinycudann.egg-info\dependency_links.txt
      writing top-level names to tinycudann.egg-info\top_level.txt
      writing manifest file 'tinycudann.egg-info\SOURCES.txt'
      reading manifest file 'tinycudann.egg-info\SOURCES.txt'
      writing manifest file 'tinycudann.egg-info\SOURCES.txt'
      copying tinycudann\bindings.cpp -> build\lib.win-amd64-cpython-38\tinycudann
      running build_ext
      error: [WinError 2] The system cannot find the file specified
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tinycudann
  Running setup.py clean for tinycudann
Failed to build tinycudann
Installing collected packages: tinycudann
  Running setup.py install for tinycudann ... error
  error: subprocess-exited-with-error

  × Running setup.py install for tinycudann did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      Building PyTorch extension for tiny-cuda-nn version 1.7
      Obtained compute capability 86 from PyTorch
      running install
      C:\Users\glens\miniconda3\envs\ns\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      running egg_info
      writing tinycudann.egg-info\PKG-INFO
      writing dependency_links to tinycudann.egg-info\dependency_links.txt
      writing top-level names to tinycudann.egg-info\top_level.txt
      reading manifest file 'tinycudann.egg-info\SOURCES.txt'
      writing manifest file 'tinycudann.egg-info\SOURCES.txt'
      running build_ext
      error: [WinError 2] The system cannot find the file specified
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> tinycudann

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

@grshennan that is tough, I think I might have run into something similar but I don't remember. If you want (this may or may not be worth it) try building tinycudann in the way suggested in this comment?
https://github.com/NVlabs/tiny-cuda-nn/issues/169#issuecomment-1365984644
Sorry I wasn't able to give you concrete help

I'm fairly new to this topic.
Would you explain, where you have placed the tiny-cuda-nn files and in which directory you have started the cmake process?
Is that necessary for everything to work?
I am receiving this error:
Traceback (most recent call last):
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\Programme\miniconda3\envs\nerfstudio\Scripts\ns-train.exe\__main__.py", line 7, in <module>
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\scripts\train.py", line 247, in entrypoint
    main(
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\scripts\train.py", line 233, in main
    launch(
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\scripts\train.py", line 172, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\scripts\train.py", line 86, in train_loop
    trainer.setup()
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\nerfstudio\engine\trainer.py", line 145, in setup
    self.pipeline = self.config.pipeline.setup(
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\nerfstudio\configs\base_config.py", line 57, in setup
    return self._target(self, **kwargs)
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\nerfstudio\pipelines\base_pipeline.py", line 229, in __init__
    self._model = config.model.setup(
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\nerfstudio\configs\base_config.py", line 57, in setup
    return self._target(self, **kwargs)
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\nerfstudio\models\base_model.py", line 82, in __init__
    self.populate_modules()  # populate the modules
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\nerfstudio\models\nerfacto.py", line 135, in populate_modules
    self.field = TCNNNerfactoField(
  File "F:\Programme\miniconda3\envs\nerfstudio\lib\site-packages\nerfstudio\fields\nerfacto_field.py", line 134, in __init__
    self.direction_encoding = tcnn.Encoding(
NameError: name 'tcnn' is not defined

It sounds like tiny-cuda-nn wasn't installed correctly. I would try running pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch again. Also try opening up a python prompt, python, then running import tinycudann to see what error you get.

It sounds like tiny-cuda-nn wasn't installed correctly. I would try running pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch again. Also try opening up a python prompt, python, then running import tinycudann to see what error you get.

Thank you very much! The reinstallation actually worked, although the pip installation method of Tiny CUDA Neural Networks has never worked before for me.
To document what I have done:

Installed CUDA 11.7 and VS 2019 and deinstalled all other versions

Created a conda envoriment
conda create --name nerfstudio -y python=3.8 conda activate nerfstudio python -m pip install --upgrade pip

Installed pytorch
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html

Git cloned tiny-cuda-nn via the repo into miniconda3\envs\nerfstudio\Lib\site-packages
git clone --recursive https://github.com/nvlabs/tiny-cuda-nn cd tiny-cuda-nn cmake . -B build cmake --build build --config RelWithDebInfo -j
(cmake commands run in VS developer command prompt)

Installed nerfstudio
pip install nerfstudio

Reinstalled tiny-cuda-nn via pip
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

I am not sure, why the installation via pip worked without errors during the last step, but nerfstudio is now fully functional.

authors: THE-COB brawlawnzo dooonnis grshennan tancik

repo: nerfstudio-project/nerfstudio


text:
rich MarkupError

Hello there,
I'm running nerfstudio on an Azure Maching Learning VM with Ubuntu 18.04.
When I try to process data with ns-process-data images I get an error which is thrown by the rich library after downscaling the images:

I already processed polycam data which worked without problems. So this error only appears in combination with colmap.
Hope you can help me and thanks in advance!

I'm a bit in a hurry to get this running, so if someone could help me out here, I would really appreciate it. :)

Can you check that your terminal is set up to use UTF-8

Can you check that your terminal is set up to use UTF-8

Yes, it is but it’s still not working.

Here is the error information
──────────────────────────────────────────────  💀 💀 💀 ERROR 💀 💀 💀  ─────────────────────────────────────────────── Error running command: colmap feature_extractor --database_path data/nerfstudio/render/colmap/database.db --image_path  data/nerfstudio/render/images --ImageReader.single_camera 1 --ImageReader.camera_model OPENCV --SiftExtraction.use_gpu 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last):   File "/home/yangtongyu/.conda/envs/nerfstudio/bin/ns-process-data", line 8, in <module>     sys.exit(entrypoint())   File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/scripts/process_data.py", line 408, in entrypoint     tyro.cli(Commands).main()   File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/images_to_nerfstudio_dataset.py", line 82, in main     self._run_colmap()   File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/colmap_converter_to_nerfstudio_dataset.py", line 199, in _run_colmap     colmap_utils.run_colmap(   File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/colmap_utils.py", line 129, in run_colmap     run_command(feature_extractor_cmd, verbose=verbose)   File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/utils/scripts.py", line 37, in run_command     CONSOLE.print(out.stderr.decode("utf-8"))   File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1673, in print     renderables = self._collect_renderables(   File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1537, in _collect_renderables     self.render_str(   File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1429, in render_str     rich_text = render_markup(   File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/markup.py", line 158, in render     print(pop_style(style_name))   File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/markup.py", line 145, in pop_style     raise KeyError(style_name) KeyError: 'home/conda/feedstock_root/build_artifacts/colmap_1686935602614/work/src/base/database.cc, line 270' (nerfstudio) yangtongyu@amax21-1:~/nerf/nerfstudio$ ns-process-data images --data data/nerfstudio/render --output-dir data/nerfstudio/render [12:02:24] 🎉 Done copying images.                                                             process_data_utils.py:241 [12:02:27] 🎉 Done downscaling images.                                                         process_data_utils.py:364 ──────────────────────────────────────────────  💀 💀 💀 ERROR 💀 💀 💀  ─────────────────────────────────────────────── Error running command: colmap feature_extractor --database_path data/nerfstudio/render/colmap/database.db --image_path  data/nerfstudio/render/images --ImageReader.single_camera 1 --ImageReader.camera_model OPENCV --SiftExtraction.use_gpu 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last):   File "/home/yangtongyu/.conda/envs/nerfstudio/bin/ns-process-data", line 8, in <module>     sys.exit(entrypoint())   File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/scripts/process_data.py", line 408, in entrypoint     tyro.cli(Commands).main()   File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/images_to_nerfstudio_dataset.py", line 82, in main     self._run_colmap()   File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/colmap_converter_to_nerfstudio_dataset.py", line 199, in _run_colmap     colmap_utils.run_colmap(   File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/colmap_utils.py", line 129, in run_colmap     run_command(feature_extractor_cmd, verbose=verbose)   File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/utils/scripts.py", line 37, in run_command     CONSOLE.print(out.stderr.decode("utf-8"))   File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1673, in print     renderables = self._collect_renderables(   File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1537, in _collect_renderables     self.render_str(   File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1429, in render_str     rich_text = render_markup(   File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/markup.py", line 161, in render     raise MarkupError( rich.errors.MarkupError: closing tag '[/home/conda/feedstock_root/build_artifacts/colmap_1686935602614/work/src/base/database.cc, line 270]' at position 13 doesn't match any open tag

Hey, is there another solution? Because my terminal supports UTF-8 but it's still not working. :(

Hi everyone, has anyone finished this problem? I found that if I use only cpu to process it will be fine (but this is too slow). Does colmap/ns-process-data can use gpu?

I'm using the official docker image and encontered the same problem. My solution is to edit line 37 of scripts.py, using try and except with print() instead of rich console if anything goes wrong. So that it prints the correct error messages.

Here is the error information ────────────────────────────────────────────── 💀 💀 💀 ERROR 💀 💀 💀 ─────────────────────────────────────────────── Error running command: colmap feature_extractor --database_path data/nerfstudio/render/colmap/database.db --image_path data/nerfstudio/render/images --ImageReader.single_camera 1 --ImageReader.camera_model OPENCV --SiftExtraction.use_gpu 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last): File "/home/yangtongyu/.conda/envs/nerfstudio/bin/ns-process-data", line 8, in <module> sys.exit(entrypoint()) File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/scripts/process_data.py", line 408, in entrypoint tyro.cli(Commands).main() File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/images_to_nerfstudio_dataset.py", line 82, in main self._run_colmap() File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/colmap_converter_to_nerfstudio_dataset.py", line 199, in _run_colmap colmap_utils.run_colmap( File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/colmap_utils.py", line 129, in run_colmap run_command(feature_extractor_cmd, verbose=verbose) File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/utils/scripts.py", line 37, in run_command CONSOLE.print(out.stderr.decode("utf-8")) File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1673, in print renderables = self._collect_renderables( File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1537, in _collect_renderables self.render_str( File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1429, in render_str rich_text = render_markup( File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/markup.py", line 158, in render print(pop_style(style_name)) File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/markup.py", line 145, in pop_style raise KeyError(style_name) KeyError: 'home/conda/feedstock_root/build_artifacts/colmap_1686935602614/work/src/base/database.cc, line 270' (nerfstudio) yangtongyu@amax21-1:~/nerf/nerfstudio$ ns-process-data images --data data/nerfstudio/render --output-dir data/nerfstudio/render [12:02:24] 🎉 Done copying images. process_data_utils.py:241 [12:02:27] 🎉 Done downscaling images. process_data_utils.py:364 ────────────────────────────────────────────── 💀 💀 💀 ERROR 💀 💀 💀 ─────────────────────────────────────────────── Error running command: colmap feature_extractor --database_path data/nerfstudio/render/colmap/database.db --image_path data/nerfstudio/render/images --ImageReader.single_camera 1 --ImageReader.camera_model OPENCV --SiftExtraction.use_gpu 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last): File "/home/yangtongyu/.conda/envs/nerfstudio/bin/ns-process-data", line 8, in <module> sys.exit(entrypoint()) File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/scripts/process_data.py", line 408, in entrypoint tyro.cli(Commands).main() File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/images_to_nerfstudio_dataset.py", line 82, in main self._run_colmap() File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/colmap_converter_to_nerfstudio_dataset.py", line 199, in _run_colmap colmap_utils.run_colmap( File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/process_data/colmap_utils.py", line 129, in run_colmap run_command(feature_extractor_cmd, verbose=verbose) File "/home/yangtongyu/nerf/nerfstudio/nerfstudio/utils/scripts.py", line 37, in run_command CONSOLE.print(out.stderr.decode("utf-8")) File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1673, in print renderables = self._collect_renderables( File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1537, in _collect_renderables self.render_str( File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/console.py", line 1429, in render_str rich_text = render_markup( File "/home/yangtongyu/.conda/envs/nerfstudio/lib/python3.8/site-packages/rich/markup.py", line 161, in render raise MarkupError( rich.errors.MarkupError: closing tag '[/home/conda/feedstock_root/build_artifacts/colmap_1686935602614/work/src/base/database.cc, line 270]' at position 13 doesn't match any open tag

same problem, seems like my output dir is within my NAS, which may goes wrong with some linux file permissions. When I change the output dir to a folder under docker /workspace/ , everything goes well.

Agree with @HugoFeng and @deephog, the issue might be coming from a database.db lock, which is based on it's location. I'm working on a remote Linux machine and ran into the issue. I moved the output folder somewhere up the directories and it worked as expected.

hey guys I got a possible solution,
If you're mounting the NAS on Linux using the mount command, make sure to toss in the nobrl option. This little trick helps avoid those annoying locking problems.
Just run something like this:
sudo mount -t cifs -o username=USERNAME,uid=$(id -u),gid=$(id -g),iocharset=utf8,nobrl //192.168.X.XX/Main /nas/Main
This should sort out those pesky errors about file permissions and database locks.

authors: HugoFeng Jordan-Pierce Terry10086 forEachWhileTrue lan-creator qf-wei tancik


text:
Make templates resolutions the Coordinate's responsibility

The puzzle 23-93ef7c09 from #23 has to be resolved:
https://github.com/nergal-perm/java-decita/blob/18bd62953fdb5dfe351db2621e3a14e4e8db6144/java-decita/src/main/java/ru/ewc/commands/SimpleCommand.java#L68-L68
The puzzle was created by Евгений Терехов on 24-May-24.
role: DEV.
If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

The puzzle 23-93ef7c09 has disappeared from the source code, that's why I closed this issue.

repo: nergal-perm/java-decita


text:
Snakefile 1  - First Production level Snakefile for  06-slurm-array/array_create_db_files.sl

Proposed Snakefile

Following Snakefile defines a configuration with the Apptainer settings and excluded files.
It gets a list of .apsimx files, excluding the specified files.
The all rule defines the final output files we want to create.
The process_apsimx rule defines how to process each .apsimx file:
It requires both the .apsimx file and the corresponding .db ( this is the placeholder .db) file as input.
It creates a .processed file as output to mark completion.

Perhaps we can modify this later to save file quota

sets the resources (memory and time) and number of threads.
The shell command loads the Apptainer module, sets the bind path, and runs the Models command.

import os
from glob import glob

# Configuration
config = {
    "apptainer_bind": "/agr/scratch,/agr/persist",
    "apptainer_image": "/agr/persist/projects/2024_apsim_improvements/apsim-simulations/container/apsim-2024.09.7579.0.aimg",
    "excluded_files": ["2023-10-09_MasterSoilApsimLibrary.apsimx", "LargerExample.apsimx"]
}

# Get list of .apsimx files
apsimx_files = [f for f in glob("*.apsimx") if f not in config["excluded_files"]]

rule all:
    input:
        expand("{file}.processed", file=[os.path.splitext(f)[0] for f in apsimx_files])

rule process_apsimx:
    input:
        apsimx = "{file}.apsimx",
        db = "{file}.db"
    output:
        "{file}.processed"
    resources:
        mem_mb = 8000,
        time = "00:10:00"
    threads: 12
    shell:
        """
        module load Apptainer
        export APPTAINER_BIND="{config[apptainer_bind]}"
        apptainer exec {config[apptainer_image]} Models {input.apsimx}
        touch {output}
        """

Configure Snakemake  - These instructions are for snakemake > 8

Create a configuration directory for Snakemake profiles:

profile_dir="${HOME}/.config/snakemake"
mkdir -p "$profile_dir"

Use cookiecutter to create the Slurm profile template:

If cookiecutter isn't installed, load snakemake module and run pip install --user cookiecutter

template="gh:Snakemake-Profiles/slurm"
cookiecutter --output-dir "$profile_dir" "$template"

During the cookiecutter process, you'll be prompted to set values for your profile. For example:

profile_name [slurm]: slurm.my_account
sbatch_defaults []: account=my_account no-requeue exclusive
cluster_sidecar_help: [Use cluster sidecar. NB! Requires snakemake >= 7.0! Enter to continue...]
Select cluster_sidecar:
1 - yes
2 - no
Choose from 1, 2 :
cluster_name []:

After completing the prompts, the profile scripts and configuration file will be installed in the $profile_dir as profile_name/[2].

You can then use this profile when running Snakemake by adding the --profile flag:

snakemake --profile slurm.my_account ...

This will configure Snakemake to submit jobs to Slurm using the settings you specified in the profile[2].
Remember that the profile typically includes scripts for job submission, status checking, and a configuration file (config.yaml) that defines default values for Snakemake command line arguments. You may need to adjust these files to match your specific Slurm environment and requirements.

Working Snakefile for this rule  : https://github.com/nesi/APSIM-HPC/pull/54

repo: nesi/APSIM-HPC


text:
Initial implementation of automatic MQTT config retrieval from HA Supervisor

Breaking change
Should be no breaking changes.
This only touches the MQTT extension.

Proposed change
The MQTT configuration should be retrieved from HA Supervisor.
I've explained the process in this issue: #1002

Type of change

[ ] Dependency upgrade
[ ] Bugfix (non-breaking change which fixes an issue)
[x ] New feature (which adds functionality to an existing integration)
[ ] Breaking change (fix/feature causing existing functionality to break)
[ ] Code quality improvements to existing code or addition of tests

Additional information

This PR fixes or closes issue: fixes #1002
This PR is related to issue: #1002
Link to documentation pull request:

Checklist

[ ] The code change is tested and works locally.
[ ] Local tests pass. Your PR cannot be merged unless tests pass
[ ] There is no commented out code in this PR.
[ ] I have followed the [development checklist][dev-checklist]
[ ] The code compiles without warnings (code quality chek)
[ ] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated for www.home-assistant.io

First draft, quickly thrown together.

You may want to make some adjustments here: https://github.com/net-daemon/netdaemon/blob/main/Docker/rootfs/etc/services.d/netdaemon_addon/run#L14-L22
Also I couldn't find if/where you define a config.yaml file for the netdaemon. I think you would want to express that the app wants rather than needs the mqtt service. HA can only provide details for it if it's installed as an addon. In my case, I don't use run mosquitto as an addon (it runs on another machine). Ideally (I think) you'd want to collect optional MQTT config from users for people who use external brokers via config.yaml and pass it through to the right env vars if it's defined in the run command above.

Consider rewrite, Mqtt env vars is already set på in run script.

repo: net-daemon/netdaemon


text:
Dependency upgrades

Breaking change

Proposed change

Type of change

[x] Dependency upgrade
[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (which adds functionality to an existing integration)
[ ] Breaking change (fix/feature causing existing functionality to break)
[ ] Code quality improvements to existing code or addition of tests

Additional information

This PR fixes or closes issue: fixes #
This PR is related to issue:
Link to documentation pull request:

Checklist

[x] The code change is tested and works locally.
[ ] Local tests pass. Your PR cannot be merged unless tests pass
[ ] There is no commented out code in this PR.
[ ] I have followed the [development checklist][dev-checklist]
[ ] The code compiles without warnings (code quality chek)
[ ] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated for www.home-assistant.io

Pull Request Test Coverage Report for Build 3183812732

0 of 0   changed or added relevant lines in 0 files are covered.
4 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.03%) to 63.208%

Files with Coverage Reduction
New Missed Lines
%

src/Daemon/NetDaemon.Daemon/Daemon/NetDaemonHost.cs
4
60.26%

Totals

Change from base Build 2943640611:
-0.03%

Covered Lines:
4505

Relevant Lines:
6666

💛  - Coveralls


text:
Ability to provide credentials

When trying to download an Instgram video the following message appears:

Provided the following link: https://www.instagram.com/reel/CybzFHyOcUo/?igshid=MzRlODBiNWFlZA==

I'd like to add on to this issue. It appears that the frontend takes links from Gallery-dl before G-DL begins downloading, which causes it to download from sources without the credentials I've manually provided to G-DL, resulting in 403 forbidden errors being downloaded as html files instead.
In my case, it was Pixiv that didn't work for all R18-tagged posts. And I don't think it would work on SW-DLT without major revision to how the frontend downloads galleries since Pixiv requires OAuth instead just providing username and password (G-DL uses an Android user-agent and Pixiv on Android browsers require OAuth afaik).

For accounts that work with username & password, you can look in the [docs[(https://github.com/net00-1/SW-DLT/blob/master/Docs.md#authentication). There's a guide to setup a-shell to keep your credentials.
I will keep this open to check for ways of other authentication types

As a newbie, do I have this directory structure set up correctly?
It keeps prompting "requires. authentication"
[~group]$ ls -Ra
.
..
.com.apple.mobile_container_manager.metadata.plist
.netrc
Library
SW_DLT.py
gallery-dl
yt-dlp
./Library:
.               Caches
..              Preferences
./Library/Caches:
.       ..
./Library/Preferences:
.
..
group.AsheKube.a-Shell.plist
./gallery-dl:
.               config.json
..
./yt-dlp:
.       ..      config

[~group]$ cat yt-dlp/config
--netrc
[~group]$ cat gallery-dl/config.json
{    "extractor": {        "twitter": {
    "username": "$$$$$$",            "password":
 "$$$$$$"        }    }}

Now much more convenient cookie-based authentication is available, you can check the updated docs about how to use it.


text:
Add standalone Seafile Docker tutorial

This PR adds a tutorial on how to set up a standalone Seafile server including a valid LetsEncrypt certificate for TLS/SSL secured HTTPS connections.
Please notice that the linked tutorial on how to install Docker and Docker-Compose is not merged yet (https://github.com/netcup-community/community-tutorials/pull/17).

I have read and understood the Contributor's Certificate of Origin at the end of the template and I hereby certify that I meet the contribution criteria described in it.
Signed-off-by: Max Kratz account@maxkratz.com

Hi maxkratz, thank you for contributing! It's been a while since you opened this PR and it probably is an open secret we're currently overwhelmed with reviewing all the awesome tutorials we got. Please  be patient - we're about to publish tutorials this and next week. Thank you!

@netcupClaudiaM Thank you for the update! Take the time you need for all the reviews :).

Hey @maxkratz, could you please upload all your pictures in english so that everything is consistent? Thanks!

@raphi-0901 No problem, I've fixed the German screenshots. Thank you for the suggestion.
Furthermore, I've added a configuration option for the services to automatically start in case the system had to be (re-)booted.

authors: maxkratz netcupClaudiaM raphi-0901

repo: netcup-community/community-tutorials


text:
android studio 2.2rc2 使用jack compiler 编译出现的问题

android studio 2.2rc2 使用jack compiler 编译出现的问题
09-09 12:51:41.657 18228-18228/com.netease.nim.demo E/AndroidRuntime: FATAL EXCEPTION: main                                                                       Process: com.netease.nim.demo, PID: 18228                                                                       java.lang.IllegalAccessError: Illegal class access: 'com.netease.nimlib.sdk.NIMClient' attempting to access 'com.netease.nimlib.r.g' (declaration of 'com.netease.nimlib.sdk.NIMClient' appears in /data/app/com.netease.nim.demo-2/base.apk)                                                                           at com.netease.nimlib.sdk.NIMClient.init(Unknown)                                                                           at com.netease.nim.demo.NimApplication.onCreate(NimApplication.java:68)                                                                           at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1015)                                                                           at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4815)                                                                           at android.app.ActivityThread.access$1600(ActivityThread.java:168)                                                                           at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1440)                                                                           at android.os.Handler.dispatchMessage(Handler.java:102)                                                                           at android.os.Looper.loop(Looper.java:150)                                                                           at android.app.ActivityThread.main(ActivityThread.java:5643)                                                                           at java.lang.reflect.Method.invoke(Native Method)                                                                           at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:799)                                                                           at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:689)
怎么样解决呢？

我现在也出现了这个，你那边解决了吗？！

@zengbobo0805
用kotlin了


text:
鼠标进入 鼠标离开消息

如果这个控件的子控件 mouse属性都是false时，
pVideoBox_->AttachMouseEnter([this](ui::EventArgs* param){
		scale_menu_box_->SetVisible(true);
		return true;
	});

	pVideoBox_->AttachMouseLeave([this](ui::EventArgs* param){

		scale_menu_box_->SetVisible(true);
		return true;
	});

要实现的，鼠标进入就显示某子控件  鼠标离开就隐藏某子控件  功能可以实现
但如果子控件mouse为true的 时候， 子控件截取了消息， 导致如上功能无法实现
我现在使用的 绘制时，通过判断鼠标位置 来动态隐藏与显示
还有更好的方式么

AttachBubbledEvent(ui::kEventMouseEnter
AttachBubbledEvent(ui::kEventMouseLeave
使用bubbledEvent

恩恩 测试成功了
这个AttachBubbledEvent   是什么意思   这样虽然有子控件，但是也能收到鼠标进入消息
是不是，这个的意思是  控件收到了什么消息 不管有没有被子控件截取


text:
怎么可以创建一个模态子窗口，然后主窗口等待子窗口的返回

模态窗口可以使用ShowModalFake函数，但如何能让主窗口阻塞在哪里，知道子窗口操作完成后，返回一个结果给主窗口呢；看了ShowMsgBox和文件对话框的实现 都没有找到好的方法

我是封装mfc对话框到dll，然后在导出的函数里doModal....

我是封装mfc对话框到dll，然后在导出的函数里doModal....

那弹出来就可以是模态对话框了吗？

我是封装mfc对话框到dll，然后在导出的函数里doModal....

那弹出来就可以是模态对话框了吗？

是的，因为我的对话框是老项目里的对话框，封到dll里就能直接用了。本质上就是写一个新的消息循环，代码就在循环里了。

authors: lsaejn yzxfz


text:
Infrastructure: bump velero version and add selector

Description
This PR bumps the version of velero, and introduces a label selector to filter out the resources originated from CrownLabs instances, in order to prevent the partial failures due to the kubevirt/velero issue.
Fixes # (issue)
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Please also note any relevant details for your test configuration.

[x] Deploying the updated manifest in the cluster, and checking that the backup completed correctly

/merge

repo: netgroup-polito/CrownLabs


text:
Fix values link on homepage

Values link was pointing to /about.

LGTM

authors: calavera hdlopes


text:
invlide update available messge

When running "netlify dev function:invoke" I got a message saying a new version of the cli was available and I should run
npm -i -g  netlify-cli
-i is invalid and should be just i
I can't find this in the source code so guess it's a message from somewhere else.

Thanks @SteveALee, it is probably coming from https://github.com/yeoman/update-notifier.
Can you please share the output of npx envinfo --system --binaries --npmPackages netlify-cli --npmGlobalPackages netlify-cli so we can try and reproduce using the same environment?

Sadly I reinstalled the cli - after having to delete some files (ntl.psi and ntl.cmd) from ...AppData/Roaming/npm/, even after npm rm -g netlify-cli
But anyway
C:\Users\steve>npx envinfo --system --binaries --npmPackages netlify-cli --npmGlobalPackages netlify-cli
npx: installed 1 in 1.639s

  System:
    OS: Windows 10 10.0.19041
    CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
    Memory: 4.08 GB / 15.86 GB
  Binaries:
    Node: 14.6.0 - C:\Program Files\nodejs\node.EXE
    npm: 6.14.6 - C:\Program Files\nodejs\npm.CMD

Did you expect netlify-cli there?
C:\Users\steve>npm ls -g netlify-cli
C:\Users\steve\AppData\Roaming\npm
`-- netlify-cli@2.59.1

Yeah I was expecting to see see the CLI, but maybe we have some installation issues with Windows machines.
Thanks for the information.

yes the update notifier text looks correct. And the version bumped .1
Perhaps I imaged the -i ?

Closing as stale. Please re-open if needed

authors: SteveALee erezrokah sarahetter

repo: netlify/cli


text:
add resource links to stages

many stages refer to links and resources, so it might be a good idea to put those resources in a list

Do you think we should separate this out in the UI, like make a text field Resource List in the stage data object?

that's probably not the worst idea, yeah

covered by #431

covered by #431

authors: jlengstorf tzmanics

repo: netlify/explorers


text:
Hidden field in editor component is rendered

- Do you want to request a feature or report a bug?
bug
- What is the current behavior?
An editor component that declares a hidden field will have it rendered:

- If the current behavior is a bug, please provide the steps to reproduce.

Declare an editor component that has a field with widget: 'hidden':

CMS.registerEditorComponent({
  // Internal id of the component
  id: "youtube",
  // Visible label
  label: "Youtube",
  // Fields the user need to fill out when adding an instance of the component
  fields: [{
    name: 'id',
    label: 'Youtube Video ID',
    widget: 'string'
  }, {
    name: 'myHiddenField',
    label: 'My hidden field',
    widget: 'hidden'
  }],
  // Pattern to identify a block as being an instance of this component
  pattern: /^youtube (\S+)$/,
  // Function to extract data elements from the regexp match
  fromBlock: function(match) {
    return {
      id: match[1]
    };
  },
  // Function to create a text block from an instance of this component
  toBlock: function(obj) {
    return 'youtube ' + obj.id;
  },
  // Preview output for this component. Can either be a string or a React component
  // (component gives better render performance)
  toPreview: function(obj) {
    return (
      '<img src="http://img.youtube.com/vi/' + obj.id + '/maxresdefault.jpg" alt="Youtube Video"/>'
    );
  }
});

Add an instance of the component to a markdown editor.

Observe that the label for the hidden field is rendered with a No control for widget: 'hidden' message below it. The same thing happens if the hidden field has no label except that a small gray box is rendered instead of the label.
- What is the expected behavior?
That the field isn't rendered, similar to how hidden widgets work for collections.

- Please mention your versions where applicable.

Netlify CMS version: 1.8.4
Browser version: Chrome 67.0.3396.62/OSX

Node.JS version: 9.5.0
Operating System: OSX
- Please link or paste your config.yml below if applicable.

Good catch!

authors: erquhart papandreou

repo: netlify/netlify-cms


text:
config should have required property 'options' - Clearstatus Netlify

I get this error post a successful build and deploy on netlify.
https://youthful-swartz-68b931.netlify.app/
Error loading the CMS configuration
Config Errors:
config should have required property 'options'
Check your config.yml file.

Hi @thelynchpin, one of your select widgets is missing the options configuration:
fields:
    - label: "Name"
    name: "name"
    widget: string
    - label: "Group"
    name: "group"
    widget: select
    multiple: false
    required: false
    - label: "Description"
    name: "description"
    widget: string
    required: false
    - label: "Link"
    name: "link"
    widget: string
    required: false

I'm new to this and have deployed the Clearstatus app through the deploy to netlify button on https://github.com/weeblrpress/clearstatus. Please specify what needs to be modified on the list.

Welcome :)
Please see the docs for the select widget here:
https://www.netlifycms.org/docs/widgets/#select
I suggest you open an issue on that repo and ask where is the config.yml file is located

@thelynchpin
Can you give it a try now, I think we fixed the problem.
@erezrokah What seems to have happened is that previously Netlify CMS would accept that select without options (which depends on the user particular site configuration) but at some point this started to trigger an error. I'm now removing the entire select if no "groups" option is configured by user.

@erezrokah What seems to have happened is that previously Netlify CMS would accept that select without options (which depends on the user particular site configuration) but at some point this started to trigger an error. I'm now removing the entire select if no "groups" option is configured by user.

Yes, we added schema validations for widgets configurations. The widget used output an error div prior to that:
https://github.com/netlify/netlify-cms/blob/519cb2d4c2db729d2643c9116f93656b6a9dba23/packages/netlify-cms-widget-select/src/SelectControl.js#L114

Thanks. It's working now.
Regards,
Wilbur
On Fri, Nov 6, 2020 at 3:13 PM clearstatus notifications@github.com wrote:

@thelynchpin https://github.com/thelynchpin
Can you give it a try now, I think we fixed the problem.
@erezrokah https://github.com/erezrokah What seems to have happened is
that previously Netlify CMS would accept that select without options (which
depends on the user particular site configuration) but at some point this
started to trigger an error. I'm now removing the entire select if no
"groups" option is configured by user.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/netlify/netlify-cms/issues/4489#issuecomment-722983208,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACEOACCGO2YD2AOUDIJEMOLSOPANZANCNFSM4SZYORPA
.

authors: clearstatus erezrokah thelynchpin


text:
[Bug]: fail build if output: 'standalone' config option is set

Summary
Those coming from the docker/k8s world are likely to follow this guidance from Next.js official docs, which instructs them to clone the with-docker example, whose config explicitly uses output: 'standalone':
module.exports = {
  output: 'standalone',
}

This will create a folder at .next/standalone which can then be deployed on its own without installing node_modules.

Additionally, a minimal server.js file is also output which can be used instead of next start. This minimal server does not copy the public or .next/static folders by default as these should ideally be handled by a CDN instead, although these folders can be copied to the standalone/public and standalone/.next/static folders manually, after which server.js file will serve these automatically.

As this relies on an always-running node.js server, it's incompatible with our serverless lyfestyle.
@ascorbic suggested that rather than changing the site's next.config.js file at build-time, we fail the build and output an error message:

I think the most useful then would be for us to detect that setting and fail the build, with an appropriate message

Having this error message will be immensely useful for those coming from Next.js wrapped in docker.
Steps to reproduce

Clone and deploy the with-docker example from Next.js examples

A link to a reproduction repository
No response
Next Runtime version
4.28.3
More information about your build

[ ] I am building using the CLI
[ ] I am building using file-based configuration (netlify.toml)

What OS are you using?
No response
Your netlify.toml file
No response
Your public/_redirects file
No response
Your next.config.js file

`next.config.js`
module.exports = {
  output: 'standalone',
}

Builds logs (or link to your logs)
No response
Function logs
No response
.next JSON files
No response

@ascorbic Is this still relevant with v5?

It won't be, because it's forced to standalone anyway

authors: ascorbic jasonbarry serhalp

repo: netlify/next-runtime


text:
update README image with new logo

Description

Removes the old version of the README image and replaces it with one donning the new Netlify logo ✨
Documentation

Tests

Visit the README file
See the change (updated Netlify logo)

Relevant links (GitHub issues, etc.) or a picture of cute animal

Superseded by v5. (I checked and it has the updated logo.)

authors: iKristy serhalp


text:
NETOBSERV-857 After some time, it fails to retrieve flows

Reload HOST token from file on every query

/ok-to-test

/lgtm
up to you to do what I suggested above or not .. this isn't super critical

/label qe-approved
Verified no longer running into the cert issue with this change when authToken: HOST is set.

@jpinsonneau are we good to merge?

/approve

authors: jotak jpinsonneau memodi


text:
DatagramDnsQueryDecoder.newQuery() raises IndexOutOfBoundsException

Actual behavior
I tried to upgrade Netty 4.1.66 to 4.1.67, but our CI tests produced the following error when resolving a DNS query.
io.netty.handler.codec.DecoderException: java.lang.IndexOutOfBoundsException: readerIndex(36) + length(2) exceeds writerIndex(36): PooledUnsafeDirectByteBuf(ridx: 36, widx: 36, cap: 2048)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:98)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:97)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: java.lang.IndexOutOfBoundsException: readerIndex(36) + length(2) exceeds writerIndex(36): PooledUnsafeDirectByteBuf(ridx: 36, widx: 36, cap: 2048)
	at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442)
	at io.netty.buffer.AbstractByteBuf.readShort(AbstractByteBuf.java:749)
	at io.netty.buffer.AbstractByteBuf.readUnsignedShort(AbstractByteBuf.java:765)
	at io.netty.handler.codec.dns.DatagramDnsQueryDecoder.newQuery(DatagramDnsQueryDecoder.java:87)
	at io.netty.handler.codec.dns.DatagramDnsQueryDecoder.decode(DatagramDnsQueryDecoder.java:64)
	at io.netty.handler.codec.dns.DatagramDnsQueryDecoder.decode(DatagramDnsQueryDecoder.java:33)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
	... 16 common frames omitted

When DatagramDnsQueryDecoder.newQuery(packet, but) was about to called, the buf had been already fully consumed by DnsMessageUtil.decodeDnsQuery().
https://github.com/netty/netty/blob/794c45fd590ffc51b16bb4b8af87556c0d2025fb/codec-dns/src/main/java/io/netty/handler/codec/dns/DatagramDnsQueryDecoder.java#L57-L64
I guess the problem is caused by the changes in https://github.com/netty/netty/pull/11415/files#diff-41d08a26f08dcb6f92085ad9bb565c833bbe0e0d425e0fe7147fcc51b56d164aR64
Minimal yet complete reproducer code (or URL to code)
https://github.com/line/armeria/blob/b5a2e1c264c264a4f504dfc1a5990866dea7b86f/core/src/test/java/com/linecorp/armeria/client/DnsMetricsTest.java#L67
Netty version
4.1.67.Final
JVM version (e.g. java -version)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)
OS version (e.g. uname -a)
Darwin  20.5.0 Darwin Kernel Version 20.5.0: Sat May  8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64 x86_64

Doh! Yeah let me fix this

@ikhoon PTAL https://github.com/netty/netty/pull/11592

authors: ikhoon normanmaurer

repo: netty/netty