text:
New native transport for kqueue

Motivation:
We currently don't have a native transport which supports kqueue https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2. This can be useful for BSD systems such as MacOS to take advantage of native features, and provide feature parity with the Linux native transport.
Modifications:

Make a new transport-native-unix-common module with all the java classes and JNI code for generic unix items. This module will build a static library for each unix platform, and included in the dynamic libraries used for JNI (e.g. transport-native-epoll, and eventually kqueue).
Make a new transport-native-unix-common-tests module where the tests for the transport-native-unix-common module will live. This is so each unix platform can inherit from these test and ensure they pass.
Add a new transport-native-kqueue module which uses JNI to directly interact with kqueue

Result:
JNI support for kqueue.
Fixes https://github.com/netty/netty/issues/2448
Fixes https://github.com/netty/netty/issues/4231

@normanmaurer @nmittler @marcuslinke @timothyjward @shpikat @caarlos0 @ninja- @rnorth @ndeloof @slandelle @headius (everyone on the related issues) - PTAL.
There is a large amount of code here ... sorry about that. However it would be nice to get some miles on it and gain more confidence.
There is also some duplication in the Channel hierarchy between EPOLL and KQUEUE. It is not possible for them to share a common hierarchy so for now I think its is OK to keep the duplication and look to improve/consolidate later.

🎉 🎉 🎉 I'll give it a try as soon as I can (will probably be next week). Thanks a lot!

Build failure seems to be related to not being able to find the artifact from a new sub-module introduced in this PR. I recall seeing this issue in the past on the CI servers, and thought we changed the build order or added an extra build pass to resolve this type of issues ... @trustin @normanmaurer do you recall?

Could not resolve dependencies for project io.netty:netty-transport-native-epoll:jar:4.1.7.Final-SNAPSHOT: Could not find artifact io.netty:netty-transport-native-unix-common:jar:linux-x86_64:4.1.7.Final-SNAPSHOT

nice to see that !

@normanmaurer @trustin - Looks like we do the following on the build servers:
$mvn clean validate
<will fail if new modules are added in the PR>
$mvn test

can we instead do:
$mvn clean install -DskipTests
$mvn test

@Scottmitch Sounds great! Will try it this weekend. Thanks for your effort!

@Scottmitch we not want to do an "install" as this will install "changes" that may break other builds on the same host.  I wonder if we can't tell maven to just resolve this somehow internally or if we can't tell maven to use an temporary local maven repository.

Let me investigate

@Scottmitch ok seems like I can use a custom local maven repos... Let me try this out on the CI and kick off another build. Also thanks for this "massive" work :)

@Scottmitch for the record the build is successful locally here :)

@Scottmitch ok it should now work on the CI as soon as the build is fixed for linux.
The problem is it builds and installs this one:
netty-transport-native-unix-common-4.1.7.Final-SNAPSHOT-linux-x86_64-fedora.jar
But it looks for:
netty-transport-native-unix-common:jar:linux-x86_64:4.1.7.Final-SNAPSHOT
The unix-common should not add -fedora to it.
Please fix and the build should work :)

@normanmaurer - I actually want the classifier on the artifact ... but I was just inconsistently using epoll.classifier and os.detected.arch inconsistently. That should now be fixed.
However it looks like the build server is still doing the clean validate ... do I have to open a new PR to get the changes?

@Scottmitch let me check the config again... maybe I missed to change it to install but at least it uses the local repository now.
Why you need to use the "-fedora" classifier ? We not used it before in the epoll transport as it works on fedora and debian etc....

Why you need to use the "-fedora" classifier

Not specifically the "-fedora" type classifier ... just the same classifier used by the epoll module (e.g. <epoll.classifier>${os.detected.name}-${os.detected.arch}</epoll.classifier>)

When trying to bootstrap my application like this:
EventLoopGroup nioEventLoopGroup = new KQueueEventLoopGroup(0,
                new DefaultThreadFactory(threadPrefix));

            bootstrap.group(nioEventLoopGroup).channel(KQueueDomainSocketChannel.class)
                    .handler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        protected void initChannel(final SocketChannel channel) throws Exception {
                            channel.pipeline().addLast(new LoggingHandler(getClass()));
                            channel.pipeline().addLast(new HttpClientCodec());
                        }
                    });

            return nioEventLoopGroup;

I get an exception
java.lang.UnsatisfiedLinkError: failed to load the required native library
	at io.netty.channel.kqueue.KQueue.ensureAvailability(KQueue.java:70)
	at io.netty.channel.kqueue.KQueueEventLoopGroup.<clinit>(KQueueEventLoopGroup.java:35)
	at com.github.dockerjava.netty.NettyDockerCmdExecFactory$UnixDomainSocketInitializer.init(NettyDockerCmdExecFactory.java:223)
	at com.github.dockerjava.netty.NettyDockerCmdExecFactory.init(NettyDockerCmdExecFactory.java:195)
	at com.github.dockerjava.core.TestDockerCmdExecFactory.init(TestDockerCmdExecFactory.java:91)
	at com.github.dockerjava.core.DockerClientImpl.withDockerCmdExecFactory(DockerClientImpl.java:161)
	at com.github.dockerjava.core.DockerClientBuilder.build(DockerClientBuilder.java:45)
	at com.github.dockerjava.client.AbstractDockerClientTest.beforeTest(AbstractDockerClientTest.java:57)
	at com.github.dockerjava.netty.exec.VersionCmdExecTest.beforeTest(VersionCmdExecTest.java:22)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:514)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:215)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
	at org.testng.TestRunner.beforeRun(TestRunner.java:656)
	at org.testng.TestRunner.run(TestRunner.java:624)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
	at org.testng.SuiteRunner.run(SuiteRunner.java:268)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
	at org.testng.TestNG.run(TestNG.java:1064)
	at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:126)
	at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:137)
	at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:58)
Caused by: java.lang.NoSuchMethodError: Method io.netty.channel.unix.Socket.recvFd(I)I not found
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1965)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1890)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1851)
	at java.lang.Runtime.load0(Runtime.java:795)
	at java.lang.System.load(System.java:1062)
	at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
	at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:255)
	at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:222)
	at io.netty.channel.kqueue.Native.loadNativeLibrary(Native.java:103)
	at io.netty.channel.kqueue.Native.<clinit>(Native.java:50)
	at io.netty.channel.kqueue.KQueue.<clinit>(KQueue.java:33)

Maybe have declared the wrong dependencies ?
<dependency>
			<groupId>io.netty</groupId>
			<artifactId>netty-transport-native-epoll</artifactId>
			<version>${netty.version}</version>
			<classifier>linux-x86_64</classifier>
			<exclusions>
				<exclusion>
					<groupId>io.netty</groupId>
					<artifactId>netty-transport-native-unix-common</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>io.netty</groupId>
			<artifactId>netty-transport-native-kqueue</artifactId>
			<version>${netty.version}</version>
			<classifier>osx-x86_64</classifier>
			<exclusions>
				<exclusion>
					<groupId>io.netty</groupId>
					<artifactId>netty-transport-native-unix-common</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>io.netty</groupId>
			<artifactId>netty-transport-native-unix-common</artifactId>
			<version>${netty.version}</version>
		</dependency>

You also need to specify the qualifier for the unix-common dependency AFAIC

Am 11.11.2016 um 08:51 schrieb marcuslinke notifications@github.com:
When trying to bootstrap my application like this:
EventLoopGroup nioEventLoopGroup = new KQueueEventLoopGroup(0,
new DefaultThreadFactory(threadPrefix));
        bootstrap.group(nioEventLoopGroup).channel(KQueueDomainSocketChannel.class)
                .handler(new ChannelInitializer<SocketChannel>() {
                    @Override
                    protected void initChannel(final SocketChannel channel) throws Exception {
                        channel.pipeline().addLast(new LoggingHandler(getClass()));
                        channel.pipeline().addLast(new HttpClientCodec());
                    }
                });

        return nioEventLoopGroup;

I get an exception
java.lang.UnsatisfiedLinkError: failed to load the required native library
at io.netty.channel.kqueue.KQueue.ensureAvailability(KQueue.java:70)
at io.netty.channel.kqueue.KQueueEventLoopGroup.(KQueueEventLoopGroup.java:35)
at com.github.dockerjava.netty.NettyDockerCmdExecFactory$UnixDomainSocketInitializer.init(NettyDockerCmdExecFactory.java:223)
at com.github.dockerjava.netty.NettyDockerCmdExecFactory.init(NettyDockerCmdExecFactory.java:195)
at com.github.dockerjava.core.TestDockerCmdExecFactory.init(TestDockerCmdExecFactory.java:91)
at com.github.dockerjava.core.DockerClientImpl.withDockerCmdExecFactory(DockerClientImpl.java:161)
at com.github.dockerjava.core.DockerClientBuilder.build(DockerClientBuilder.java:45)
at com.github.dockerjava.client.AbstractDockerClientTest.beforeTest(AbstractDockerClientTest.java:57)
at com.github.dockerjava.netty.exec.VersionCmdExecTest.beforeTest(VersionCmdExecTest.java:22)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86)
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:514)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:215)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:142)
at org.testng.TestRunner.beforeRun(TestRunner.java:656)
at org.testng.TestRunner.run(TestRunner.java:624)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
at org.testng.SuiteRunner.run(SuiteRunner.java:268)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
at org.testng.TestNG.run(TestNG.java:1064)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:126)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:137)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:58)
Caused by: java.lang.NoSuchMethodError: Method io.netty.channel.unix.Socket.recvFd(I)I not found
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1965)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1890)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1851)
at java.lang.Runtime.load0(Runtime.java:795)
at java.lang.System.load(System.java:1062)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:255)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:222)
at io.netty.channel.kqueue.Native.loadNativeLibrary(Native.java:103)
at io.netty.channel.kqueue.Native.(Native.java:50)
at io.netty.channel.kqueue.KQueue.(KQueue.java:33)
Maybe have declared the wrong dependencies ?

io.netty
netty-transport-native-epoll
${netty.version}
linux-x86_64

io.netty
netty-transport-native-unix-common

io.netty
netty-transport-native-kqueue
${netty.version}
osx-x86_64

io.netty
netty-transport-native-unix-common

io.netty
netty-transport-native-unix-common
${netty.version}

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@normanmaurer Using classifier  'osx-x86_64' I get the same exception.

@marcuslinke  - netty-transport-native-unix-common is a transitive dependency that you shouldn't have to explicitly include in your application. This is where all the unix general stuff from the transport-native-epoll module moved to so it could be shared with transport-native-kqueue. The goal is that you should be able to include the same way you do the epoll module http://netty.io/wiki/native-transports.html#using-the-native-transport.
While its in PR form make sure you do the following:
$ cd netty
$ git checkout <this branch>
$ mvn clean install -DskipTests

And your netty.version property should be set to 4.1.7.Final-SNAPSHOT.
Let me know if there are still issues after trying this.

@Scottmitch The problem is that I've to support both systems (linux and macos), so how this should work. As I understand the os-maven-plugin detects the build systems OS so the dependencies classifier is set to 'macos' effectively. Therefore I decided to hardcode the classifiers in the POM somehow. Any ideas?

@Scottmitch Why netty-transport-native-unix-common needs a qualifier? Is it really needed?

The problem is that I've to support both systems (linux and macos), so how this should work.

Depends how you want to achieve this.

isolate all your epoll or kqueue code in a module that is conditionally declared as a dependency in the pom.
Always include them both, and just check KQueue.isAvailable() or EPOLL.isAvailable() before you use either.

OS so the dependencies classifier is set to 'macos' effectively

Perhaps but is their a reasons to hardcode and not just use the recommended approach?

netty-transport-native-unix-common also builds a native library ... although it is static and included in the kqueue's and epoll's native library at build time. It may be possible to omit the clarifier I would prefer not to as it clarifies which platform the artifact was built on, and incase there needs to be any differences between platforms we can accommodate.

@Scottmitch Your branch compiles and installs fine here. But even when I do as suggested I get the same exception...
java.lang.UnsatisfiedLinkError: failed to load the required native library

io.netty
netty-transport-native-epoll
4.1.7.Final-SNAPSHOT
linux-x86_64

io.netty
netty-transport-native-kqueue
4.1.7.Final-SNAPSHOT
osx-x86_64

@Scottmitch You said the native library libnetty-unix-common.a is used only at compile time of the epoll and kqueue modules. Maybe there is something wrong in my local build so this library is linked dynamically and missing at runtime?

Maybe there is something wrong in my local build

Its possible ... thanks for helping me test on different environments 👍
If you run the following all the tests pass, or do you see the same issue?
$ cd transport-native-unix-common-tests
$ mvn clean install
$ cd ../transport-native-unix-common
$ mvn clean install
$ cd ../transport-native-kqueue
$ mvn clean install

I have a build failure in transport-native-unix-common:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile (default-compile) on project netty-transport-native-unix-common: Compilation failure: Compilation failure:
[ERROR] /Users/marcus/git/netty/transport-native-unix-common/src/main/java/io/netty/channel/unix/PeerCredentials.java:[21] error: cannot find symbol
[ERROR] symbol:   static EMPTY_INTS
[ERROR] location: class
[ERROR] /Users/marcus/git/netty/transport-native-unix-common/src/main/java/io/netty/channel/unix/PeerCredentials.java:[40,35] error: cannot find symbol

please run a mvn clean install -DskipTests at the root level of the repo and then try again.

[INFO] Netty/Transport/Native/Unix/Common ................. SUCCESS [  2.608 s]
[INFO] Netty/Transport/Native/Unix/Common/Tests ........... SUCCESS [  1.550 s]
[INFO] Netty/Transport/Native/KQueue ...................... SUCCESS [ 43.670 s]
[INFO] Netty/All-in-One ................................... SUCCESS [  9.113 s]
[INFO] Netty/Tarball ...................................... SUCCESS [  0.209 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:06 min
[INFO] Finished at: 2016-11-11T21:41:46+01:00
[INFO] Final Memory: 72M/871M

And still java.lang.UnsatisfiedLinkError: failed to load the required native library when try to use kqueue from within my project.

java.lang.NoClassDefFoundError: io/netty/channel/unix/LimitsStaticallyReferencedJniMethods here https://github.com/Scottmitch/netty/blob/218592882316d5a070f7d9245fcb08954e32ec5b/transport-native-kqueue/src/main/java/io/netty/channel/kqueue/KQueue.java#L35

When doing the following:
$ cd transport-native-unix-common-tests
$ mvn clean install
$ cd ../transport-native-unix-common
$ mvn clean install
$ cd ../transport-native-kqueue
$ mvn clean install

I get:
Running io.netty.channel.kqueue.KQueueSocketChannelNotYetConnectedTest
21:55:40.819 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
21:55:40.828 [Time-limited test] DEBUG i.n.c.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 16
21:55:40.844 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
21:55:40.845 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
21:55:40.845 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
21:55:40.846 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
21:55:40.846 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
21:55:40.846 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
21:55:40.847 [Time-limited test] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
21:55:40.847 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
21:55:40.847 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
21:55:40.847 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
21:55:40.848 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
21:55:40.929 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent - Javassist: available
21:55:40.929 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /var/folders/y3/8mx9s87n01942z74s5wjsxjw0000hb/T (java.io.tmpdir)
21:55:40.929 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
21:55:40.929 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
21:55:40.930 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent - io.netty.maxDirectMemory: 3817865216 bytes
21:55:40.943 [Time-limited test] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
21:55:40.943 [Time-limited test] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
21:55:40.944 [Time-limited test] DEBUG i.n.util.internal.PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available
21:55:40.963 [Time-limited test] DEBUG i.n.u.internal.NativeLibraryLoader - -Dio.netty.tmpdir: /var/folders/y3/8mx9s87n01942z74s5wjsxjw0000hb/T (java.io.tmpdir)
21:55:40.963 [Time-limited test] DEBUG i.n.u.internal.NativeLibraryLoader - -Dio.netty.native.workdir: /var/folders/y3/8mx9s87n01942z74s5wjsxjw0000hb/T (io.netty.tmpdir)
21:55:40.979 [Time-limited test] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv4Stack: false
21:55:40.979 [Time-limited test] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv6Addresses: false
21:55:40.982 [Time-limited test] DEBUG io.netty.util.NetUtil - Loopback interface: lo0 (lo0, 0:0:0:0:0:0:0:1)
21:55:40.983 [Time-limited test] DEBUG io.netty.util.NetUtil - /proc/sys/net/core/somaxconn: 128 (non-existent)
dyld: lazy symbol binding failed: Symbol not found: _clock_gettime
  Referenced from: /private/var/folders/y3/8mx9s87n01942z74s5wjsxjw0000hb/T/libnetty-transport-native-kqueue4723678605595081288.dylib (which was built for Mac OS X 10.12)
  Expected in: /usr/lib/libSystem.B.dylib
dyld: Symbol not found: _clock_gettime
  Referenced from: /private/var/folders/y3/8mx9s87n01942z74s5wjsxjw0000hb/T/libnetty-transport-native-kqueue4723678605595081288.dylib (which was built for Mac OS X 10.12)
  Expected in: /usr/lib/libSystem.B.dylib
/bin/sh: line 1: 83323 Trace/BPT trap: 5       /Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/bin/java -server -dsa -da -ea:io.netty... -XX:+AggressiveOpts -XX:+TieredCompilation -XX:+UseBiasedLocking -XX:+UseFastAccessorMethods -XX:+OptimizeStringConcat -XX:+HeapDumpOnOutOfMemoryError '-javaagent:/Users/marcus/.m2/repository/org/mortbay/jetty/alpn/jetty-alpn-agent/2.0.4/jetty-alpn-agent-2.0.4.jar=${jetty.alpnAgent.option}' -D_ -D_ -D_ -jar /Users/marcus/git/netty/transport-native-kqueue/target/surefire/surefirebooter830902336434320850.jar /Users/marcus/git/netty/transport-native-kqueue/target/surefire/surefire219842681247456731tmp /Users/marcus/git/netty/transport-native-kqueue/target/surefire/surefire_06621318527175539997tmp

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

After reading https://github.com/Homebrew/homebrew-core/issues/4809 I've installed command line tools from Apple. The Symbol not found: _clock_gettime issue disappears then but now there is another problem: https://gist.github.com/marcuslinke/24ec140def4cd162cd6180aba34ca171

@marcuslinke - Can you update your JDK? There was a known issue in your JDK related to GCM ciphers that would core dump.

@Scottmitch I've updated my JDK to 1.8.0_111 and the kqueue related tests now run successfully. But still having the issue when try to use the kqueue stuff in my own project:
Tests run: 5, Failures: 1, Errors: 0, Skipped: 4, Time elapsed: 1.495 sec <<< FAILURE! - in com.github.dockerjava.netty.exec.VersionCmdExecTest
beforeTest(com.github.dockerjava.netty.exec.VersionCmdExecTest)  Time elapsed: 1.388 sec  <<< FAILURE!
java.lang.UnsatisfiedLinkError: failed to load the required native library
	at com.github.dockerjava.netty.exec.VersionCmdExecTest.beforeTest(VersionCmdExecTest.java:22)
Caused by: java.lang.NoClassDefFoundError: io/netty/channel/unix/LimitsStaticallyReferencedJniMethods
	at com.github.dockerjava.netty.exec.VersionCmdExecTest.beforeTest(VersionCmdExecTest.java:22)
Caused by: java.lang.ClassNotFoundException: io.netty.channel.unix.LimitsStaticallyReferencedJniMethods
	at com.github.dockerjava.netty.exec.VersionCmdExecTest.beforeTest(VersionCmdExecTest.java:22)

For the files: my project is compiled with Java 7 (1.7.0_80)

After adding dependency to unix-common I ended up again with:
Caused by: java.lang.NoSuchMethodError: Method io.netty.channel.unix.Socket.recvFd(I)I not found
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

@marcuslinke - Is your project open source? Is https://github.com/docker-java/docker-java the project you are attempting to use kqueue in? If so can you send me a link to a PR so I can poke around?

Sure, here it is https://github.com/docker-java/docker-java/tree/netty-native-kqueue

Just run mvn clean install -Dit.test=VersionCmdExecTest

@marcuslinke - The test fails, but I think it is because I don't have docker running (or someone opening the /var/run/docker.sock socket). I was able to accomplish this with some changes to your pom. I commented out the epoll module for now because I didn't go through the trouble of building the epoll module and copying it to my local maven repo. However when we publish artifacts we could publish epoll built from MacOS and kqueue built from Linux to make including in the pom files easier (the other alternative is to partition you project as described in https://github.com/netty/netty/pull/6000#issuecomment-260015816). Let me know if this allows to you to do testing.
Test output:
09:37:32.123 INFO  c.g.d.c.AbstractDockerClientTest - ======================= BEFORETEST =======================
09:37:32.126 INFO  c.g.d.c.AbstractDockerClientTest - Connecting to Docker server
09:37:32.498 DEBUG c.g.d.core.command.AbstrDockerCmd - Cmd: busybox,com.github.dockerjava.netty.exec.InspectImageCmdExec@2b506a79
Tests run: 5, Failures: 1, Errors: 0, Skipped: 4, Time elapsed: 1.097 sec <<< FAILURE! - in com.github.dockerjava.netty.exec.VersionCmdExecTest
beforeTest(com.github.dockerjava.netty.exec.VersionCmdExecTest)  Time elapsed: 1.02 sec  <<< FAILURE!
java.lang.RuntimeException: socket not found: /var/run/docker.sock
	at com.github.dockerjava.netty.exec.VersionCmdExecTest.beforeTest(VersionCmdExecTest.java:22)

diff --git a/pom.xml b/pom.xml
index 39f84a4..441ef78 100644
--- a/pom.xml
+++ b/pom.xml
@@ -244,35 +244,17 @@
 			<artifactId>netty-handler-proxy</artifactId>
 			<version>${netty.version}</version>
 		</dependency>
-		<dependency>
+		<!--<dependency>
 			<groupId>io.netty</groupId>
 			<artifactId>netty-transport-native-epoll</artifactId>
 			<version>${netty.version}</version>
-			<classifier>linux-x86_64</classifier>
-			<exclusions>
-				<exclusion>
-					<groupId>io.netty</groupId>
-					<artifactId>netty-transport-native-unix-common</artifactId>
-				</exclusion>
-			</exclusions>
-		</dependency>
+			<classifier>${os.detected.classifier}</classifier>
+		</dependency>-->
 		<dependency>
 			<groupId>io.netty</groupId>
 			<artifactId>netty-transport-native-kqueue</artifactId>
 			<version>${netty.version}</version>
-			<classifier>osx-x86_64</classifier>
-			<exclusions>
-				<exclusion>
-					<groupId>io.netty</groupId>
-					<artifactId>netty-transport-native-unix-common</artifactId>
-				</exclusion>
-			</exclusions>
-		</dependency>
-		<dependency>
-			<groupId>io.netty</groupId>
-			<artifactId>netty-transport-native-unix-common</artifactId>
-			<version>${netty.version}</version>
-			<classifier>osx-x86_64</classifier>
+			<classifier>${os.detected.classifier}</classifier>
 		</dependency>
 		<dependency>
 			<groupId>junit</groupId>
@@ -308,7 +290,7 @@
 			<extension>
 				<groupId>kr.motd.maven</groupId>
 				<artifactId>os-maven-plugin</artifactId>
-				<version>1.4.0.Final</version>
+				<version>1.5.0.Final</version>
 			</extension>
 		</extensions>
 		<pluginManagement>

@Scottmitch Thanks a lot! So it seems the epoll module SNAPSHOT dependency retrieved from the main 4.1 branch has caused this.

@Scottmitch Some of my tests that passed with the old epoll module now fail with throwing io.netty.util.IllegalReferenceCountException when using the kqueue module. For example:
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:110)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:98)
	at io.netty.buffer.AbstractPooledDerivedByteBuf.deallocate(AbstractPooledDerivedByteBuf.java:73)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:115)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:98)
	at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:112)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:72)
	at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:1)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:233)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:435)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:250)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:233)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
	at io.netty.channel.kqueue.AbstractKQueueStreamChannel$KQueueStreamUnsafe.readReady(AbstractKQueueStreamChannel.java:609)
	at io.netty.channel.kqueue.KQueueDomainSocketChannel$KQueueDomainUnsafe.readReady(KQueueDomainSocketChannel.java:127)
	at io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe.readReady(AbstractKQueueChannel.java:335)
	at io.netty.channel.kqueue.KQueueEventLoop.processReady(KQueueEventLoop.java:201)
	at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:273)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	at java.lang.Thread.run(Thread.java:745)

Do you have an idea what could cause this?

@marcuslinke - Do you mean that running this PR with transport-native-epoll on linux doesn't report an exception, but running this PR with transport-native-kqueue on MacOS does report this exception?
Based on at io.netty.buffer.AbstractPooledDerivedByteBuf.deallocate(AbstractPooledDerivedByteBuf.java:73) being in the stack trace this exception may be caused by the same issue as https://github.com/netty/netty/issues/5999. Can you use the UnpooledByteBufAllocator until this PR is pulled in?

Do you mean that running this PR with transport-native-epoll on linux doesn't report an exception, but running this PR with transport-native-kqueue on MacOS does report this exception?

Right.

Can you use the UnpooledByteBufAllocator until the fix for #5999 is pulled in?

Not sure how to do this. My custom handlers in the pipeline only allocate buffers via Unpooled.buffer(). Running with -Dio.netty.allocator.type=unpooled makes the issue disappear.

@marcuslinke thanks for reporting back... Using the system property is for sure a good way to change the default.

@normanmaurer So I think changing the system property indicates a netty issue here. Right?

@marcuslinke yes... This is what @Scottmitch referred to with  #5999.

@normanmaurer Thank. But how to deal with this PR? @Scottmitch Will the fix for #5999 be merged into #6000?

@marcuslinke nope... once #5999 is fixed and merged we will rebase this pr and you can test again. Stay tuned, @Scottmitch is working on a fix

As @normanmaurer said I will push a PR which fixed #5999 soon :) (but this is an independent issue)

Running with -Dio.netty.allocator.type=unpooled makes the issue disappear.

Thanks for confirming! Do you have any other issues or is everything working as expected?

My custom handlers in the pipeline only allocate buffers via Unpooled.buffer()

Unrelated to this issue, but it is recommended to use the allocator configured in the Bootstrap object. This can be accessed through the Channel or the ChannelHandlerContext

@Scottmitch Beside issue #5999 everything works as expected. You did a great job. Thanks again! Hope this will be merged soon.

@marcuslinke - Yay! Thanks for your help!
Before pulling this in I will polish the netty-all deployment story first ... and hopefully get some more volunteers ( @slandelle @vietj 😉) to get a bit more confidence

@Scottmitch Whats the current state of this PR? Will it be merged in the near future?

@marcuslinke yes after 4.1.8.Final.

I cloned the PR and got Unix Domain Sockets running on my Mac via kqueue!  Thank you guys for your great work :)
I ran into an issue during shutdown though.  I'm not sure if it is because I am using Netty incorrectly, please forgive my ignorance if that is the case.
Our shutdown logic for a Unix domain socket server is:
eventLoop.shutdownGracefully();
try {
    channel.closeFuture().sync();
}
catch (InterruptedException e) {
    Thread.currentThread().interrupt();
}

where eventLoop is of type KQueueEventLoopGroup and channel is of type KQueueServerDomainSocketChannel.
This same logic worked for Epoll running on Linux.  But when I tried the above with kqueue, it hangs indefinitely on the sync() part of channel.closeFuture().sync();.

@dannyferreira - Can you try and reverse the order in which you shutdown the event loop and close the channel? The channel may need to use the event loop for shutdown operations.

build failure captured in issue https://github.com/netty/netty/issues/6577

@Scottmitch - closing the channel before shutting down the event loop seems to work well for both kqueue and epoll.  Thanks!

@normanmaurer - Any objections to merging this? I'm not sure that anyone has actually reviewed the code. We need to coordinate the release cycle and builds on the CI servers.
Release Cycle
I'm not very familiar with how the maven release process works but IIRC we need to do something like the following:

On linux: mvn clean install

either publish available artifacts directly from this machine, or copy artifacts over the machine in step (2))

On Mac: mvn clean install -Duber

If we released in step (1) we should only have to build/release transport-native-kqueue, all, and tarball.

It doesn't matter which platform is built first but the -Duber will build netty-all and depends upon all native libraries being available.  IIRC we do something similar for the release of tcnative which includes jars for all the different platforms? We should be able to do something similar here.
CI server PR builds
We need to build PRs on MacOS and Linux (to some degree). It would be nice if we could just build the kqueuebits on MacOS and build the rest on Linux (since we have more Linux CI servers). I don't have much visibility into the CI configuration so I'm not sure how to accomplish this.

@Scottmitch please see the comment: https://github.com/Scottmitch/netty/pull/1#issuecomment-290704233

@Scottmitch Managed to build and run your kqueue transport and run it with docker java but seems it fails on one of my colleagues machine (he didn't build your branch but just reused the jar):
dyld: lazy symbol binding failed: Symbol not found: _netty_unix_util_parse_package_prefix
Referenced from: /private/var/folders/dc/_328f1bn5zggz0xm2ztxxfmr0000gn/T/libnetty-transport-native-kqueue4910433518549296650.dylib
Expected in: flat namespace
Any idea ?

4.1 (3cc405296)

@lucboutier - What version of macOS was the artifact built on, and what version of macOS are you trying to use the artifact on? Now that the PR has been merged can you open an issue?

@Scottmitch Tried to use kqueue native transport under MacOS 10.11.6 and got
dyld: lazy symbol binding failed: Symbol not found: _clock_gettime
  Referenced from: /private/var/folders/y3/8mx9s87n01942z74s5wjsxjw0000hb/T/libnetty-transport-native-kqueue8841250350785055197.dylib (which was built for Mac OS X 10.12)
  Expected in: /usr/lib/libSystem.B.dylib

Any idea if this could be fixed somehow?

Created an issue for it: https://github.com/netty/netty/issues/6837

authors: Scottmitch dannyferreira lucboutier marcuslinke normanmaurer slandelle tejksat vietj


text:
JdkSslContext supported cipher suites incorrect

Motivation:
JdkSslContext builds the list of supported cipher suites, but assumes that ciphers prefixed with SSL_ and TLS_ will be interchangeable. However this is not the case and only applies to a small subset of ciphers. This results in the JdkSslContext attempting to use unsupported ciphers.
Modifications:

When building the list of ciphers in JdkSslContext we should first check if the engine supports the TLS_ prefix cipher.

Result:
Fixes https://github.com/netty/netty/issues/7673

4.0 (8eb80a35e0)

@Scottmitch can you also note the sha for 4.1 ?

@normanmaurer - GitHub notes the sha above as part of the merge ... am I the only one that can see this bcz I merged it?

@Scottmitch ah yeah I did not see this 🤣

source: gharchive/pull-request

authors: Scottmitch normanmaurer

repo: netty/netty

license_type: permissive

license_source: github-api


text:
light-4j benchmark

light-4j benchmark scores seem to have fallen
https://www.techempower.com/benchmarks/#section=data-r17&hw=ph&test=plaintext
vs
https://www.techempower.com/benchmarks/#section=data-r15&hw=ph&test=plaintext
time for some fixing?
https://github.com/TechEmpower/FrameworkBenchmarks/issues?utf8=✓&q=light-4j

@Dmole If you look at the source code there, you will realize nothing changed but the performance numbers are down significantly. I think they have changed the environment. In my opinion, these benchmarks are not close to real business scenarios. Our strength is https/2 with client-side service discovery. We really need to have a service to service benchmark comparing gateway, sidecars, and light-4j.

I checked and there was a 600% drop with no environment change. There are plenty of other http/2 frameworks on that list.
My best guess at the moment is that the java change when they moved to docker is the cause.... but it's just a guess.
https://github.com/TechEmpower/FrameworkBenchmarks/issues/4218

@Dmole Thanks for bringing it up to the attention of TechEmpower team. I think it is a configuration issue and we just need to compare how Undertow is configured in round 16 and later. The performance should be the same.

source: gharchive/issue

authors: Dmole stevehu


text:
期望实现encodeWithCoder 和 initWithCoder 里面的代码

期望实现encodeWithCoder 和 initWithCoder 里面的代码
如：

(void)encodeWithCoder:(NSCoder *)aCoder
{
[aCoder encodeObject:self.title forKey:@"title"] ;
[aCoder encodeObject:self.iconName forKey:@"iconName"];
[aCoder encodeObject:self.iconGrayName forKey:@"iconGrayName"];
[aCoder encodeObject:self.moduleNum forKey:@"moduleNum"];
[aCoder encodeObject:self.moduleBounceID forKey:@"moduleBounceID"];
[aCoder encodeBool:self.isCanVisit forKey:@"isCanVisit"];
//    [aCoder encodeObject:self.didClickModuleBlock forKey:@"didClickModuleBlock"];
}

(instancetype)initWithCoder:(NSCoder *)aDecoder
{
self = [super init] ;
if(self)
{
self.title = [aDecoder decodeObjectForKey:@"title"] ;
self.iconName = [aDecoder decodeObjectForKey:@"iconName"];
self.iconGrayName = [aDecoder decodeObjectForKey:@"iconGrayName"];
self.moduleNum = [aDecoder decodeObjectForKey:@"moduleNum"] ;
self.moduleBounceID = [aDecoder decodeObjectForKey:@"moduleBounceID"] ;
self.isCanVisit = [aDecoder decodeBoolForKey:@"isCanVisit"] ;
}
return self ;
}

现在主流json开源库都有api自动对模型进行kvc，并且由于复杂json如果这样一行一行推起来，模型文件估计很难看所以还是用第三方json库里的api吧，本人的json解析WHC_Model，SexyJson，有兴趣可以去了解一下

好的，谢谢

authors: KateSS netyouli

license: MIT


text:
Serialization issues with large dictionary

Hi,
first of all, thanks for MsgPack and ZeroFormatter.
today I tested ZeroFormatter to serialize a dictionary and write the array to a file.
First tested it with a small set and all worked fine. Was able to serialize to binary and save to a file, read the file en deserialize back to the dict. All great.
But now that I want to process the actual dataset, the serializer takes like forever. The dictionary contains about 4.5 million rows with key and value being a custom class.
Is there a limit to the amount of rows for a dictionary to work smoothly?
here a sample of my code:
        public void ZeroFormatterSerializeToFile()
        {
            try
            {
                if(MsgEvent != null)
                    MsgEvent.Invoke(this, MethodBase.GetCurrentMethod().Name + " Start serializing");

                var bytes = ZeroFormatterSerializer.Serialize(dict);

                if(MsgEvent != null)
                    MsgEvent.Invoke(this, MethodBase.GetCurrentMethod().Name + " Serializing done. Start write to file");

                File.WriteAllBytes(_serializeFileName ?? "ZF_" + _defaultFileName, bytes);

                if(MsgEvent != null)
                    MsgEvent.Invoke(this, MethodBase.GetCurrentMethod().Name + " End write to file");
            }
            catch(Exception ex)
            {
                ex = ex;
                throw ex;
            }
        }

It's taking over 20 minutes now and counting :-( (in release by the way)
Thanks,
Matthijs
Processor	Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz, 3201 Mhz, 6 Core(s), 12 Logical Processor(s)
Installed Physical Memory (RAM)	32.0 GB
OS Name	Microsoft Windows 10 Pro
.Net 4.8

Resolved the problem with this code:
    ```

public void ZeroFormatterSerializeToFileTest()
{
try
{
if(MsgEvent != null)
MsgEvent.Invoke(this, MethodBase.GetCurrentMethod().Name + " Start serializing");
            using(FileStream fs = File.Create("test" + (_serializeFileName ?? "ZF_" + _defaultFileName)))
            {
                using(var bw = new BinaryWriter(fs))
                {
                    foreach(var item in dict)
                    {
                        var rowBytes = ZeroFormatterSerializer.Serialize(item);
                        bw.Write(rowBytes.Length);
                        bw.Write(rowBytes);
                    }
                }
            }

            if(MsgEvent != null)
                MsgEvent.Invoke(this, MethodBase.GetCurrentMethod().Name + " Serialization done and End write to file");
        }
        catch(Exception ex)
        {
            ex = ex;
            throw ex;
        }
    }

    public void ZeroFormatterDeSerializeFromFileTest()
    {
        try
        {
            using(BinaryReader reader = new BinaryReader(File.Open("test" + (_serializeFileName ?? "ZF_" + _defaultFileName), FileMode.Open)))
            {
                int length = (int)reader.BaseStream.Length;
                while(reader.BaseStream.Position != length)
                {
                    int bytesToRead = reader.ReadInt32();
                    byte[] rowBytes = reader.ReadBytes(bytesToRead);
                    var result = ZeroFormatterSerializer.Deserialize<KeyValuePair<K, V>>(rowBytes);
                    if(result.Key != null && result.Key != null)
                        dict[result.Key] = result.Value;
                }
            }
        }
        catch(Exception ex)
        {
            throw ex;
        }
    }

is this best practice?
regards,

Matthijs

@mldz can you fix the formatting on your post, thanks

authors: jzabroski mldz

repo: neuecc/ZeroFormatter

license: mit

license_source: bigquery


text:
Introduce dict[str, Performance]

Related to #491
This PR introduces following changes:

Remove metric_name member from Performance.
Change all list[Performance] to dict[str, Performance], whose keys are metric names.
Refactoring related to these changes.

Originally, metric_name is used as a primary key of the list, and often the list is converted to dict in the library. This change attempts to removes this burden.

One high-level comment that could be considered as future PRs.

refactoring metric_configs, overall and  analyses to dict[str, ...] as well

authors: odashi pfliu-nlp

repo: neulab/ExplainaBoard


text:
Update the example_synthetic files

Once https://github.com/neurobagel/annotation_tool/issues/597 is closed, update the example_synthetic files using the files in the annotation tool under cypress/fixtures/examples/good

@alyssadai Aside from example_synthetic.json having to be replaced with the one from annotation tool, and adding the example_synthetic_expected_output.json is there anything else that needs to updated?

Hey @rmanaem,
To avoid having to change the file names in the other tools that use this example set (e.g., the CLI), here's my suggestion:

Add https://github.com/neurobagel/annotation_tool/blob/master/cypress/fixtures/examples/good/example_synthetic.json to the data-upload/ folder but give it a different name, such as example_synthetic_participants.json  since it's meant to represent the pre-Neurobagel BIDS participants.json for this dataset
Replace example_synthetic.json in this repo with example_synthetic_expected_output.json but keep the same original name

Let me know if that makes sense!

Sounds good!

authors: alyssadai rmanaem


text:
Turn on merge queues

The overhead of dependabot PRs is mainly having to wait for them to go back to green after the previous one merged.
Github auto-merge can help, because it will automatically merge PRs once all requirements are met.
Tasks

[x] identify the workflows that need to (re-) run in a merge queue
[ ] update these workflows with the merge_group event trigger
[ ] manually update non-synced workflows in these repos

[ ] https://github.com/neurobagel/query-tool
[ ] https://github.com/neurobagel/federation-api
[ ] https://github.com/neurobagel/api
[ ] https://github.com/neurobagel/bagel-cli

Something to look into:

[x] if I approve two bot PRs, does the CI for the second one run again after the first one was merged?
[ ] if I allow an external PR to auto-merge, and then the author makes accepts some code suggestions from me, do I have to re-enable the merge / merge manually

edit:
old and incomplete view:
That's what merge queues are for (and hopefully that works here to): https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue
So let's turn them on and let GitHub manage the merge sequence

I think I misunderstood this partly. What I am looking for is more "auto-merging": https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/automatically-merging-a-pull-request
Merge queues are also nice, but they address a problem that we don't have yet: how to merge a lot of PRs one after the other, if the CI for each PR takes a lot of time.
I think we can still enable merge queues if we want to (we now have them in https://github.com/neurobagel/react-bagel) but this issue should be about whether we want auto-merges turned on.
For us auto-merge would mean:

unless you explicitly disable it on your own PR, the PR will by default merge as soon as all checks are green, you are approved, and all other requirements are met

For external contributors it would mean:

we need to enable auto-merge for their PRs
and then the PR auto merges as well, unless they make another commit

Some things we've learned already:

auto-merge runs once all checks and requirements are met
however: auto-merge does not rerun checks if the target branch has been pushed since the last check, and it also doesn't auto-merge / update PR branches

So for these things, you'd still want a merge queue

https://medium.com/@kojoru/how-to-set-up-merge-queues-in-github-actions-59381e5f435a

Shared workflows we want to run on merge queues:

none

Repo specific workflows we want to run on merge queues

query

[ ] codespell (should be synced)
[ ] component test
[ ] e2e test -> these should be both in one workflow and they should use coverage like in our react template
[ ] lint
[ ] prettier -> they should both be in one workflow and also synced to all react repos!
[ ] codespell (see above)

n-api

[ ] codespell (see above)
[ ] test

f-API

[ ] codespell (see above)
[ ] lint -> should be synced to all python repos
[ ] test

cli

[ ] codespell (see above)
[ ] test

moving this issue to tracked until we address the following workflows issues:

[ ] neurobagel/workflows#90
[ ] neurobagel/workflows#91
[ ] neurobagel/workflows#92
[ ] neurobagel/workflows#93

Addressing these issues will also implicitly resolve the merge_queue issue but by making one change instead of 4 or 5

Closing as we don't (yet) have very long running checks

repo: neurobagel/planning


text:
Qa tractography pr

Add qa_tractography.py in stats, and changing the dwi_pipeline.

You also need to add more information to your tutorial, you don't explain what inputs your qa takes, what is happening inside of the program, and what to expect from the results

repo: neurodata/ndmg


text:
Bug when using Sentry form Matlab call

when calling SCT from Matlab, if Sentry is set, the following error is sent (for any SCT script):
    import raven
  File "/Users/jcohen/code/sct/python/lib/python2.7/site-packages/raven/__init__.py", line 54, in <module>
    from raven.base import *  # NOQA
  File "/Users/jcohen/code/sct/python/lib/python2.7/site-packages/raven/base.py", line 37, in <module>
    from raven.conf.remote import RemoteConfig
  File "/Users/jcohen/code/sct/python/lib/python2.7/site-packages/raven/conf/remote.py", line 36, in <module>
    DEFAULT_TRANSPORT = discover_default_transport()
  File "/Users/jcohen/code/sct/python/lib/python2.7/site-packages/raven/conf/remote.py", line 18, in discover_default_transport
    from raven.transport.threaded import ThreadedHTTPTransport
  File "/Users/jcohen/code/sct/python/lib/python2.7/site-packages/raven/transport/__init__.py", line 13, in <module>
    from raven.transport.eventlet import *  # NOQA
  File "/Users/jcohen/code/sct/python/lib/python2.7/site-packages/raven/transport/eventlet.py", line 12, in <module>
    from raven.transport.http import HTTPTransport
  File "/Users/jcohen/code/sct/python/lib/python2.7/site-packages/raven/transport/http.py", line 14, in <module>
    from raven.utils.http import urlopen
  File "/Users/jcohen/code/sct/python/lib/python2.7/site-packages/raven/utils/http.py", line 11, in <module>
    import ssl
  File "/Users/jcohen/code/sct/python/lib/python2.7/ssl.py", line 97, in <module>
    import _ssl             # if we can't import it, let the error propagate
ImportError: dlopen(/Users/jcohen/code/sct/python/lib/python2.7/lib-dynload/_ssl.so, 2): Symbol not found: _SSL_CTX_set_alpn_protos
  Referenced from: /Users/jcohen/code/sct/python/lib/python2.7/lib-dynload/_ssl.so
  Expected in: /Applications/MATLAB_R2016a.app/bin/maci64/libssl.1.0.0.dylib
 in /Users/jcohen/code/sct/python/lib/python2.7/lib-dynload/_ssl.so
Total processing time: 0 min 0 s

When turning off Sentry at installation, this error is fixed.

Matlab is bundling its own libssl and prepending its dynamic library path to even the subprocesses it calls, which is wrong. We can work around that in the script launcher if this is a common usage pattern.

It is not a common usage pattern (for now, and i hope it will not become one), so turning off Sentry is an acceptable workaround. Good to know about the cause though.

authors: jcohenadad zougloub

repo: neuropoly/spinalcordtoolbox


text:
Running sessions "lost" if xrdp-sesman restarts

If xrdp restarts then the new process is not aware of any sessions created by the old process, so you can't re-attach to them and they aren't listed by xrdp-sesadmin.  Would it be possible for the new process to "discover" them and populate g_sessions?
Test case:

Connect to a new xorgxrdp (Xorg) session.
xfreerdp /v:localhost /u:<user> /p:<password> &

List sessions using xrdp-sesadmin.  The session created above will be listed.
xrdp-sesadmin -u=<user> -p=<password> -c=list

Restart xrdp.  Note that the existing sessions are not disconnected.
sudo service xrdp restart

List sessions using xrdp-sesadmin.
xrdp-sesadmin -u=<user> -p=<password> -c=list
Expected behaviour: the session created above should be listed.
Observed behaviour: "No sessions." (or, prior to #797, "Error getting session list.")

Disconnect the RDP client (without logging out) from the RDP session.

Attempt to reconnect to the existing session.
xfreerdp /v:localhost /u:<user> /p:<password> &

(Workaround: to reconnect to the "orphaned" sessions you can set up xvnc to view the appropriate X display.)
Note that the new xrdp would somehow have to obtain the session_item info from each session it discovers.  Also the new xrdp wouldn't be the parent of the sesman for each session so it wouldn't receive SIGCHLD when the session ends (i.e. it would have to poll).  I would suggest solving both of these using a shared mmap() object.

Actually, running session will be lost when xrdp-sesman daemon restarts. Restarting only xrdp daemon won't lost session info.

Thanks, yes!  I have corrected the issue title.

I've corrected the title again :)

Oops, thanks!
To elaborate on my suggestion above, xrdp-sesman could open a file say /var/run/xrdp-sesman.sessions and mmap it into memory.  It can store each struct session_item (from the g_sessions linked list) in an array in that region rather than allocated by malloc().
Now if xrdp-sesman restarts, the new process can mmap the file and discover the existing sessions.  Using mmap has the advantage that the contents can be shared with other processes, so it can be used to communicate between the main xrdp-sesman instance and its session instances - for example, to update the user's idle time.  (A shared pthread mutex can be used to synchronise access.)
To determine if a session instance of xrdp-sesman is still running the main instance can no longer listen for SIGCHLD; instead each session could have a "heartbeat" timestamp that it updates in its mmap struct once a minute.  If that timestamp is a few minutes out of date then we can assume that the session has ended.  (Of course, if the pid is gone or has the wrong name then we can also assume that the session has ended, but checking shared memory is easier than checking /proc.  There are some other ways to detect other processes exiting but they aren't portable.)

Ben, I think that if xrdp-sesman restarts it will halt all sessions running
under it anyway
On Wed, Jul 12, 2017 at 11:02 PM Ben Cohen notifications@github.com wrote:

Oops, thanks!
To elaborate on my suggestion above, xrdp-sesman could open a file say
/var/run/xrdp-sesman.sessions and mmap it into memory. It can store each struct
session_item (from the g_sessions linked list) in an array in that region
rather than allocated by malloc().
Now if xrdp-sesman restarts, the new process can mmap the file and
discover the existing sessions. Using mmap has the advantage that the
contents can be shared with other processes, so it can be used to
communicate between the main xrdp-sesman instance and its session instances

for example, to update the user's idle time. (A shared pthread mutex can
be used
https://gist.github.com/ben-cohen/67d41a917ffbf7a5da99fd622feb346b to
synchronise access.)

To determine if a session instance of xrdp-sesman is still running the
main instance can no longer listen for SIGCHLD; instead each session could
have a "heartbeat" timestamp that it updates in its mmap struct once a
minute. If that timestamp is a few minutes out of date then we can assume
that the session has ended. (Of course, if the pid is gone or has the wrong
name then we can also assume that the session has ended, but checking
shared memory is easier than checking /proc. There are some other ways
http://bewareofgeek.livejournal.com/2945.html to detect other processes
exiting but they aren't portable.)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/neutrinolabs/xrdp/issues/800#issuecomment-314880512,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADTH1KBsQQs6kTtEnuWKbO-1loEbShl_ks5sNSZDgaJpZM4OP6DQ
.

--
Idan Freiberg
PGP FP: 8108 7EC9 806E 4980 75F2  72B3 8AD3 2D04 337B 1F18

Overall, the feature is nice and I'm very interested in :)

@speidy - That doesn't happen in practice.  If it was intended to do that then this issue is fairly pointless!  Is there a reason why it should kill the sessions on exit or is it just that it can't yet re-discover them?
Test case:

Connect and log in.
rdesktop localhost &

Stop xrdp.
sudo service xrdp restart

Observed: the xrdp session is still connected and running.  (But the new xrdp-sesadmin doesn't know about it as described above.)
The sessions are not killed because:

When the daemon instance starts it sets g_pid to the current process id, and installs signal handlers including sig_sesman_shutdown() for SIGINT and SIGTERM.  These are inherited by the session instances when they are forked, and not changed thereafter.

service xrdp restart calls /etc/init.d/xrdp which uses start-stop-daemon to send SIGINT to the xrdp-sesadmin daemon instance.  It doesn't send anything to the session instances.

The daemon instance handles SIGINT using sig_sesman_shutdown().  That calls session_sigkill_all(), which sends SIGTERM to each of the session instances.

The session instance handles SIGTERM using sig_sesman_shutdown().  But the process id doesn't match g_pid, which is the daemon instance's process id, so it returns without doing anything (except unhelpfully logging "shutting down sesman 1"), but because the signal was handled it continues running.

Maybe you right and thats not happens when xrdp-sesman is running in deamon
mode / background mode.
When I run sesman in foreground mode and then send sigterm to it, all the
sessions being closed.
I'll test later if you scenario actually works as you expect.
On Thu, Jul 13, 2017 at 7:54 PM Ben Cohen notifications@github.com wrote:

@speidy https://github.com/speidy - That doesn't happen in practice. If
it was intended to do that then this issue is fairly pointless! Is there a
reason why it should kill the sessions on exit or is it just that it can't
yet re-discover them?
Test case:

Connect and log in.
rdesktop localhost &
2.
Stop xrdp.
sudo service xrdp restart
Observed: the xrdp session is still connected and running. (But the new
xrdp-sesadmin doesn't know about it as described above.)
The sessions are not killed because:

When the daemon instance starts it sets g_pid to the current process
id, and installs signal handlers including sig_sesman_shutdown() for SIGINT
and SIGTERM. These are inherited by the session instances when they are
forked, and not changed thereafter.
2.
service xrdp restart calls /etc/init.d/xrdp which uses
start-stop-daemon to send SIGINT to the xrdp-sesadmin daemon instance. It
doesn't send anything to the session instances.
3.
The daemon instance handles SIGINT using sig_sesman_shutdown(). That
calls session_sigkill_all(), which sends SIGTERM to each of the session
instances.
4.
The session instance handles SIGTERM using sig_sesman_shutdown(). But
the process id doesn't match g_pid, which is the daemon instance's process
id, so it returns without doing anything (except unhelpfully logging
"shutting down sesman 1"), but because the signal was handled it continues
running.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/neutrinolabs/xrdp/issues/800#issuecomment-315137947,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADTH1Dyc-bFnTaOGRMoIJZW0IvYi566_ks5sNkvLgaJpZM4OP6DQ
.

--
Idan Freiberg
PGP FP: 8108 7EC9 806E 4980 75F2  72B3 8AD3 2D04 337B 1F18

@ben-cohen I think your'e right.
Thats actually only happens in foreground mode, where you press Ctrl+C so signal being sent to the forks (sessions) as well.
I like that feature and idea.
+1

@speidy I've just come to the same conclusion about Ctrl+C sending SIGINT to the whole process group.  Thanks, I'll try it then...

I have created PR #819 for this.  Let me know if you find any bugs or have any comments.

I discovered today that the systemd unit files for xrdp.service and xrdp-sesman.service on Ubuntu 20.04 bind the two services together. There doesn't seem to be a way to restart xrdp without also restarting xrdp-sesman on that platform, short of rewriting the unit files.
Any updates on this issue? I was going to reply to @matt335672 in https://github.com/neutrinolabs/xrdp/pull/819, but wanted to avoid adding more comments to a possibly dead PR.

Hi @arnonerba
I'll split your question into two bits; firstly, this issue under discussion, and secondly a possible way forward for your particular Ubuntu issue.
As regards the current issue, there's quite a lot to do before we get near this.
The current plan is to revamp and simplify all the inter-process comms used by xrdp. At the moment I'm looking at the link between xrdp and xrdp-sesman. The output from this work will allow us to split sesman into two bits - a session header process for each session, and an overall controller. The headers will maintain virtually all the state for each session connection, and the controller will then be little more than a broker which can easily be restarted. It's a similar model to the one used by ssh. When the controller is restarted, any necessary state info can be passed back up from the session headers to the controller.
I'd initially started at the other end, as I was hoping for a quick fix to #1684 which is relatively important. However, this turned out to be a bad approach. Solving the harder problems first is likely to end up with a better result in the long run.
I'm happy to talk more about this. I'm aware I may not have explained the above very well. We are looking at it, but as with most projects we're having to keep a lot of plates spinning at the same time with fairly limited resources.
As regards the Ubuntu unit files, you don't have to re-write these - you can provide overrides pretty easily straightforwardly which do what you want. See systemd.unit(5) an the details on the edit sub-command in systemctl(1) for more info.
Getting the Ubuntu system files changed, requires talking to the Ubuntu team. As a downstream project, our links for communication with Ubuntu/CentOS, etc are the same as everyone else's - raising a fault on Launchpad (Ubuntu) or Bugzilla (CentOS). Don't expect a quick response to this process!
Is that useful information?

Thanks, @matt335672! I appreciate the quick and detailed response. That makes a lot of sense, and it's good to hear that this is still being worked on along with everything else.
I'll think about opening a bug on Launchpad. Do you think it makes sense to treat xrdp.service and xrdp-sesman.service completely separately?

Personally I think the services should be separate, as xrdp doesn't need sesman in some configurations. However, there's a complication which Ubuntu has brought on themselves for what are actually very good reasons.
The Ubuntu packaging guys have set up the xrdp service to run as a non-privileged user, which is a much more secure way of doing things. However, this means they have to jump through a couple more hoops to get the services started. There's a bit of info on this on the wiki.
From what I found when I looked into this, there's no reason why the services need to be coupled, but I could be wrong.
If you're interested in this, I suggest you try editing the service unit descriptions and see how it goes. If you get something useful, please add the changes you've made to this PR as well as raising them in Launchpad. That will be very useful for the rest of the user community.
Let me know if you need a hand with any of this.

Hi Matt,
Thanks for the extra analysis. I'm realizing that I'm not going to have the chance to experiment with the unit files anytime soon. If I can find some time to work on this, I'll be sure to report back here.

authors: arnonerba ben-cohen matt335672 metalefty speidy

repo: neutrinolabs/xrdp


text:
Typo fixes and more

I used a more sophisticated script to run the sources through the spell checker while suppressing uninteresting stuff (hex constants, include file names etc), which allowed me to catch more typos. Some changes are not strictly typo fixes, so they have been split into separate patches.

+1


text:
NV6145 and put a timeout mechanism on the pathWalkers

(1) To prevent pathWalkers from running into an indefinite state.
(2) For debugging purpose, the enforcer will pass the container identifier to the pathWalker, too.

Added:
NV6145: Ability to disable specific functionality to reduce resource utilization
New environmental variable to disable the secret scans on container: "ENF_NO_SECRET_SCANS=1"

NV6145: Add more debug parameters to enforcer side support log.
When the env variable "ENF_MONITOR_TRACE=1" is set, the extra pathWalker debug traces will be included in the enforcer logs.

repo: neuvector/neuvector


text:
Question regarding Ida-gen branch

Are there plans to support generating ida type definitions, or typelibs?

The current branch was made as an experiment. At the moment it is not a priority, but as soon as we have time, we will do it. But there is no ETA

Elaborating further, at some point, we gave up on keeping the ida-gen branch up2date, and as of now, it lacks a lot of fixes/other stuff. While implementing all the things that would be needed for IDA generation, some functionalities were stripped and implemented in a very sketchy way(it was intended to be a POC after all). There are a lot of improvements that could(should) be made within the current codebase, at some point we will design a proper way how to opt the generator and will add the IDA support there.


text:
Update to Pharo 7 up to 12 and Tonel

@newapplesho

the PR includes an update for the Twilio Client to Pharo 7, Pharo 8, Pharo 9. Pharo 10. Pharo 11, Pharo 12
it also adds many tests and a CI group (for using with Smalltalk CI and GitHub Workflows)
the branch is now called "main"
it uses a proper "src" directory

@astares
Thank you very much.
I would like to make some changes and would like to use this as a reference to make the modifications in another branch, is that OK?

@newapplesho Yes, totally! Thanks for checking it.

https://github.com/newapplesho/twilio-smalltalk/pull/5

authors: astares newapplesho

repo: newapplesho/twilio-smalltalk


text:
golang版本的ipad协议在这里

https://github.com/wechat-robot/wechat-golang

404了哥


text:
Fprats new golden metrics

Relevant information
Modified and added files for summary_metrics and golden_metrics for the following entities:
GCPSPANNERINSTANCE
GCPSTORAGEBUCKET
GCPAPPENGINESERVICE
GCPBIGQUERYDATASET
GCPBIGQUERYTABLE
GCPCLOUDFUNCTION
GCPCLOUDSQL
GCPHTTPLOADBALANCER
GCPINTERNALLOADBALANCER
Checklist

[ ] I've read the guidelines and understand the acceptance criteria.
[ ] The value of the attribute marked as identifier will be unique and valid.
[ ] I've confirmed that my entity type wasn't already defined. If it is I'm providing an
explanation above.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.fprats seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

missing license agreement

authors: CLAassistant fprats


text:
There is no certificate created yet

Hey guys, we tried to run nr1 nerdpack:serve but we got this error
There is no certificate created yet.
Error: An error ocurred while performing the operation
    at CallocService._buildRequest (/usr/local/lib/newrelic-cli/node_modules/@datanerd/3pp-cli-utils/src/ServiceFacade.js:149:23)

But I didn't find anything related, could you help us to fix this?
Thanks

@rv-vsilva have you created a profile first? this is not a cloud-optimize specific error, but something to do with the cli configuration.

Thanks! Yes... I'm trying to validate if the provided key is working. I will close the issue as this is not related.


text:
[CI/CD] lint, test, release chart

[x] Moving e2e charts to /.charts
[x] adapt pipelines
[x] Test implementation in fork

I rebased and moved around things, but the changes with respect to the old pipeline are just:
index 872250d..c2a4585 100644
--- a/.github/workflows/release-integration.yml
+++ b/.github/workflows/release-integration.yml
@@ -84,7 +84,7 @@ jobs:
       - name: Push pre-release images
         if: ${{ github.event.release.prerelease }}
         run: |
-          echo "DOCKER_IMAGE_TAG=${DOCKER_IMAGE_TAG}-pre" >> $GITHUB_ENV
+          DOCKER_IMAGE_TAG=${DOCKER_IMAGE_TAG}-pre"
           docker buildx build --push --platform=$DOCKER_PLATFORMS \
             -t $DOCKER_IMAGE_NAME:$DOCKER_IMAGE_TAG \
             .
diff --git a/charts/e2e-resources/Chart.yaml b/charts/e2e-resources/Chart.yaml
index ef96b38..49e879c 100755
--- a/charts/e2e-resources/Chart.yaml
+++ b/charts/e2e-resources/Chart.yaml
@@ -1,6 +1,5 @@
 apiVersion: v2
-version: 1.2.0
-appVersion: "1.0.0"
+version: 1.2.0-devel
 description: This chart creates e2e resources for nri-kubernetes.
 name: e2e-resources

waiting for the ci to kick in, to merge this

repo: newrelic/nri-kubernetes


text:
Perhaps the install command could be improved

Steps to recreate:

Have an app with an existing newrelic.yml
Run newrelic install with any options

newrelic.yml file already exists.  Move it out of the way.

https://github.com/newrelic/rpm/blob/17734c85a9251ede14fbee517f0d264d99116c7e/lib/new_relic/cli/commands/install.rb#L44
The language could be less blunt, and also this could give the option of overwriting the file instead.

Hi @flanger001, we'll look into the improvement you've suggested - certainly the wording could be kinder, I agree! Thanks for bringing it up.

Thanks @rachelleahklein! I think the more useful thing here would be my second point of optionally overwriting the file. I think it's probably a safe assumption that this file will be in VC.

@flanger001 Thank you for your suggestion.  We have improved newrelic install with a --force flag and improved messaging.  This will be available in the next release after 6.9.0.

Thank you @mwlang!

authors: flanger001 mwlang rachelleahklein

repo: newrelic/rpm


text:
CI: use -fstack-protector-all

Just a little change inspired by #1598. See the commit message for details.
Reviews are welcome! I'll merge in three days.

Coverage remained the same at 58.663% when pulling 0e15740c7b757687ad62b115d452a79fa741cd3a on feature/ci-use-stack-protector-all into 469931933b0de3de1225680e09bcdad341fee514 on master.

authors: Minoru coveralls

repo: newsboat/newsboat


text:
Include tags in OPML feed export (fixes #871)

OPML, while primarily using a hierarchy of outlines, also supports tags in version 2.0 (http://opml.org/spec2.opml#1629042387000):

category is a string of comma-separated slash-delimited category strings, in the format defined by the RSS 2.0 category element. To represent a "tag," the category string should contain no slashes. Examples: 1. category="/Boston/Weather". 2. category="/Harvard/Berkman,/Politics".

By exporting as OPML 2.0 we can include the tags in a way other programs understand. Commas are replaced with underscores. Slashes are left as-is, so when newsboat imports a hierarchical OPML file and then exports it again, that hierarchy isn't lost.
Fixes #871

Coverage: 59.55% (+0.2%) from 59.376% when pulling c88a29fd314fb3029bb8ddabaca1e16f34ce6ff7 on gwymor:opml-tags into 7299d6b8ff7c081d59743f874857f663fc0da6a4 on newsboat:master.

Thank you for the pull request, and sorry it took me so long to take a look!
I like the approach, but I think we should introduce it in a backward-compatible manner:

adjust the code so it can generate either OPML 1.0 or OPML 2.0;
make -e/--export-to-opml generate OPML 1.0 (i.e. keep the behaviour that existed before this PR). Adjust the help text for that option to clearly state what version it produces;
add --export-to-opml2 which, well, exports in OPML 2.0 format :)
file an issue for 3.0 release where we should make -e an alias for --export-to-opml2, and simultaneously introduce --export-to-opml1 for backward compatibility.

I should probably create an issue about importing 2.0 too, because it'd be weird for Newsboat to be unable to read the files it produced.

In other words, would any software or workflows be broken if we introduce 2.0 without a separate option?

I don't have the energy to properly compare the specs right now, but I think that if the two versions were compatible, the newer one would have been called 1.1. So I'm just playing it safe and asking for a separate option. If we discover that the two specs are indeed compatible (at least, compatible enough for our purposes), we can alias -e to --export-to-opml2 before Newsboat 3.0, no problem.
Thank you very much for taking the time to update the docs too. It's rare contributor who cares that much.
Anyway, this looks awesome and it'll get auto-merged once Coveralls re-caluclated the test coverage. Congratulations on your first contribution to Newsboat!

authors: Minoru coveralls gwymor


text:
The query for affected and fixed packages should be abstracted as methods on the models (and/or on a model manager)

The query for affected and fixed packages should be abstracted as methods on the models (and/or on a model manager)
Originally posted by @pombredanne in https://github.com/nexB/vulnerablecode/pull/525#discussion_r791658135
Currently we query for affected and fixed packages as: https://github.com/nexB/vulnerablecode/blob/67fc00e5a8dbdf690f1c1b0d627da0d06322adb9/vulnerabilities/views.py#L81-L87
This needs to be abstracted as model methods.
Related:

https://github.com/nexB/vulnerablecode/issues/596

@Hritik14 @pombredanne  .. is this issue open for contribution ?

Hi @Hritik14 and @pombredanne, can I work on this issue, or has it already been solved?

We now have these at least in https://github.com/nexB/vulnerablecode/blob/457f8f8ce8550502714d3f19d914c5ab2409c935/vulnerabilities/models.py#L651-L670

affected_by
fixing

Closing this now. Note that we still need to rework the models as described in https://github.com/nexB/vulnerablecode/issues/966

We now have these at least in https://github.com/nexB/vulnerablecode/blob/457f8f8ce8550502714d3f19d914c5ab2409c935/vulnerabilities/models.py#L651-L670

affected_by
fixing

Closing this now. Note that we still need to rework the models as described in https://github.com/nexB/vulnerablecode/issues/966

repo: nexB/vulnerablecode


text:
[주지민] step2 - 서비스 배포하기

step1에서 확인해주셨던 내부망 3306은 외부망에게만 허용되게 수정했습니다!
NAT 게이트웨이 이용해서 내부망 연결을 추가해봤는데, 처음써봐서 문제가 없는지.. 궁금합니다

리뷰 감사합니다!!

NAT 게이트웨이 이용해서 내부망 연결을 추가해봤는데, 처음써봐서 문제가 없는지.. 궁금합니다
내부망이 인터넷과 통신해야하는 이유가 있었을까요?

NAT 게이트웨이 이용해서 내부망 연결을 추가해봤는데, 처음써봐서 문제가 없는지.. 궁금합니다
내부망이 인터넷과 통신해야하는 이유가 있었을까요?

과제로 올려주신 운영 DB의 도커 이미지 받아오기위해서 필요하다고 생각했습니다!

repo: next-step/infra-subway-deploy


text:
🛑 /getMusicFeed - Get music feed by URL or ID is down

In 380c790, /getMusicFeed - Get music feed by URL or ID (https://tiktok.nextpost.tech/getMusicFeed?query=7129970471625362182&cache_timeout=0&license_key=$API_KEY) was down:

HTTP code: 400
Response time: 4217 ms

Resolved: /getMusicFeed - Get music feed by URL or ID is back up in 1b7389e after 5 minutes.

repo: nextpost-tech/tiktok-status


text:
[BUG] - Setting tabs content with flex does not work

NextUI Version
2.6.8
Describe the bug
I wanted to set the Tabs title width to 1/3 of the parent element and the content to 2/3, so I added flex-1 to base classname and flex-2 to panel. The base works, but the panel doesn't seem to work. You can see flex-2 in the classname of the panel but not work in webpage
Your Example Website or App
No response
Steps to Reproduce the Bug or Issue
Init nextui with nextui-cli and then use the code
import { Tabs, Tab } from "@nextui-org/react";
import React from "react";

class TABStest extends React.Component {

    render() {
        return

            <Tabs
                isVertical={true}
                classNames={{
                    base: 'flex-1',
                    tabList: 'min-h-screen ',
                    panel: 'flex-2',
                }}
                fullWidth={true}>
                <Tab key="1" title="TAB1">

                        Content1

                </Tab>
                <Tab key="2" title="TAB2">

                        Content2

                </Tab>
                <Tab key="3" title="TAB3">

                        Content3

                </Tab>
                <Tab key="4" title="TAB4">

                        Content4

                </Tab>
            </Tabs>

    }
}
export default TABStest

Expected behavior
Set the content width and works
Screenshots or Videos
No response
Operating System Version
windows
Browser
Chrome

This issue doesn't relate to NextUI but to Tailwindcss. Your approach is actually correct. The reason it didn't work is because flex-2 is not a part of tailwindcss utilities (available options: flex-1, flex-auto, flex-initial and flex-none. The classname is there but you can see flex is not in Styles section in inspector. However, you can customise flex-2 by adding the following to your taiwind.config.js.
theme: {
  extend: {
    flex: {
      '2': '2 2 0%'
    }
  }
}

In this way, you should see the following

authors: MYmyRedustY wingkwong

repo: nextui-org/nextui


text:
Feat/v2

Closes #
📝 Description

Add a brief description

⛳ Current behavior (updates)

Please describe the current behavior that you are modifying

🚀 New behavior

Please describe the behavior or changes this PR adds

💣 Is this a breaking change (Yes/No):

📝 Additional Information

Hey, does this mean that NextUI is moving away from Stitches to Tailwind? If so, will there be any good migration path for existing users? This seems like a huge change, that would mean rewriting almost all styling code.
Interested to hear what your thoughts are on this!

I am very excited about this

authors: hmbrg jrgarciadev twsau


text:
pipeline does not generate cnv models

Description of the bug
The pipeline ends with error with ~120 samples:
 A USER ERROR has occurred: Bad input: Filtering removed all intervals.  Select less strict filtering criteria.
However it does not generate cnv model or ploidy (a run in cohort mode with the samples provided). Only readcounts are available in folder readcounts (hdf5 format).
Command used and terminal output
nextflow run /playground/nf-core-createpanelrefs_dev/dev \
   -profile singularity \
   --input /playground/dataset_cnvs/samplesheets_sarek/samplesheet_bams_illumina.csv \
   --tools germlinecnvcaller \
   --fasta /corebm/tmp/bams/hg38.fa --dict /corebm/tmp/bams/hg38.dict --fai /corebm/tmp/bams/hg38.fa.fai \
   --outdir /playground/dataset_cnvs/germlinecnvcaller_illumina_reference_nochrM \
   --ploidy_priors /playground/dataset_cnvs/priors.table \
   --padding 250 \
   --bin_length 0 \
   --mappable_regions /playground/dataset_cnvs/Illumina-truseq-rapid-exome_v1.2_hg38_target_orig_nochrM.bed

Relevant files
next.log
System information
nextflow version 23.10.1.5891
Workstation
local executor
singularity
ubuntu 22.04
nf-core/createpanelrefs dev (22/01/2024)

Hi @egenomics Could you run the latest dev and see if it works now?

authors: egenomics ramprasadn

repo: nf-core/createpanelrefs


text:
Add salmon quantification to salmon_alignment_base subworkflow

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Module is added. Parameters for STAR_AlIGN are updated

repo: nf-core/dualrnaseq


text:
pipeline has no release, but no UNDER CONSTRUCTION warning

Check Documentation
I have checked the following places for your error:

[ ] nf-core website: troubleshooting
[ ] nf-core/gwas pipeline documentation

Description of the bug

Steps to reproduce
Steps to reproduce the behaviour:

Command line:
See error:

Expected behaviour

Log files
Have you provided the following extra information/files:

[ ] The command used to run the pipeline
[ ] The .nextflow.log file

System

Hardware:
Executor:
OS:
Version

Nextflow Installation

Version:

Container engine

Engine:
version:
Image tag:

Additional context

The About section has now been updated. Closing this issue now.

repo: nf-core/gwas


text:
Thank other companies/services supporting nf-core

HTML is probably bad practise, feel free to fix ;)

@nf-core-bot fix linting

Couldn't resist tidying up the HTML a little, sorry. Also switched a couple of the logos to use a horizontal layout, plus a Docker logo that works better on dark mode.


text:
feat(pager): add align to sides

#208

Merged, thnx! In the future please squash commits into one and add Fixes #xyz to indicate which issue is being fixed by the commit.

authors: pkozlowski-opensource sendilkumarn

repo: ng-bootstrap/core


text:
[question] i18n file changes are not taken into account (spa hmr)

I'm submitting a ...  (check one with "x")
[x] bug report => check the README and search github for a similar issue or PR before submitting
[ ] support request => check the README and search github for a similar issue or PR before submitting
[ ] feature request

Current behavior

When adding keys to i18n files, changes are not taken into account. This works well on universal server (dev + prod) but not with spa.
Thanks  !
Expected/desired behavior

Keys should be translated when using {{'KEY' | translate}} and not showing KEY.
Minimal reproduction of the problem with instructions

modify an i18n file
clean cache, clean everything (even this step should not be needed)
load spa

Quick fix : clear every browser cache. But this should not happen in dev.

@theomathieubhvr I suppose you're using HMR on spa build and I have exactly the same issue right here.
I digged it a little and found out that's more or less connected with copy-assets-webpack-plugin and webpack-dev-server.
It's a known issue with the HMR; files copied through copy-assets-webpack-plugin are not taken into account.
Rebuilding the project using npm run build:spa-dev and then running npm run serve:spa would definitely solve this problem.
On the other hand, if you're using ngx-cache then the app seeks the cache (by default, should be local-storage) for i18n data. And yeah, clearing the cache (or just local-storage) would solve it.
Please feel free to to add anything if there're any points that I missed.

That's exactly it !
But in production, if I update my i18n keys, how can I force my users to reload their localstorage ?
Thanks !

You can set TTL (in terms of secs, ex: 300) right at the imports section of you app.module. I suppose there's no usage example on ngx-cache/core repository about TTL but see the cache.loader.ts to what params to pass. Also, don't forget to set the expiry param to undefined - otherwise it will override the TTL.

repo: ng-seed/universal


text:
change publicPath to point to CDN

[ ] Support request =>
[ ] Feature request
[x ] Documentation issue or request

**Current behavior**
When editing the publicPaths field of the config it always prepends with the HOST.

**Expected/desired behavior**
How can I allow the publicPaths to point to a CDN url?

#507


text:
Fix typo on readme

PRIVILEGES not PRIVIELGES

Merged into master. Thanks @James-TR

authors: James-TR ngoduykhanh

repo: ngoduykhanh/PowerDNS-Admin


text:
Question about the generate_predictions() function

When I run the demo.ipynb file without changing anything and try getting the autotuned predictions, gpt3 works fine, but once I use gpt4 and the promptcast model, I get this error:

TypeError                                 Traceback (most recent call last)
Cell In[9], line 6
4 hypers = list(grid_iter(model_hypers[model]))
5 num_samples = 2
----> 6 pred_dict = get_autotuned_predictions_data(train, test, hypers, num_samples, model_predict_fns[model], verbose=False, parallel=False)
7 out[model] = pred_dict
8 plot_preds(train, test, pred_dict, model, show_samples=True)
File /mnt/aamv_data/nimeesha_workspace/nimeesha_workspace/first_paper/AAMV/llmtime/models/validation_likelihood_tuning.py:119, in get_autotuned_predictions_data(train, test, hypers, num_samples, get_predictions_fn, verbose, parallel, n_train, n_val)
117     best_val_nll = float('inf')
118 print(f'Sampling with best hyper... {best_hyper} \n with NLL {best_val_nll:3f}')
--> 119 out = get_predictions_fn(train, test, **best_hyper, num_samples=num_samples, n_train=n_train, parallel=parallel)
120 out['best_hyper']=convert_to_dict(best_hyper)
121 return out
File /mnt/aamv_data/nimeesha_workspace/nimeesha_workspace/first_paper/AAMV/llmtime/models/promptcast.py:278, in get_promptcast_predictions_data(train, test, model, settings, num_samples, temp, dataset_name, **kwargs)
275 input_strs = None
276 if num_samples > 0:
277     # Generate predictions
--> 278     preds, completions_list, input_strs = generate_predictions(model, inputs, steps, settings, scalers,
279                                                                 num_samples=num_samples, temp=temp, prompts=prompts, post_prompts=post_prompts,
280                                                                 parallel=True, return_input_strs=True, constrain_tokens=False, strict_handling=True, **kwargs)
281     # skip bad samples
282     samples = [pd.DataFrame(np.array([p for p in preds[i] if p is not None]), columns=test[i].index) for i in range(len(preds))]
TypeError: models.promptcast.generate_predictions() got multiple values for keyword argument 'parallel'
Changing parallel=False to True, or removing the parameter in the function call altogether doesn't work. What should I do?
Thank you!

I received the same error.

authors: NimeeshaChan nickreich

repo: ngruver/llmtime


text:
Possible to send options/arguments to Redux Devtools extension?

Is there any way to pass options/arguments to the redux-devtools-extension?  It seems that maxAge gets carried along, but it would be nice to be able to specify the other options as well.

I second that. Would be really useful

authors: JohnKis loverde

repo: ngrx/store-devtools


text:
audio wave view issue when activity open again

step by step

open audio list activity
selet audio and open audio trim activity ( work good )
but when press back so goto audio select activity
now select audio again and open audio trim activity
audio wave view not render

https://user-images.githubusercontent.com/55321122/112315828-848f4480-8cd0-11eb-8af7-f2e0fe76ac54.MOV

same issue when debug app and open first time

My demo is working good. I think you should check view when inititalize. You need call setAudioPath function after view initialized. You need call it inside post function because when activity just showed up, view's width and height are zero
audioView.post {
audioView.setAudioPath(...)
}

problem solve
audioWaveView.post(new Runnable() {
                @Override
                public void run() {
                    audioWaveView.setAudioPath(srcFile);
                }
            });


text:
Open MatDialog from button in an ngx-formly form

How to add a button in a ngx-formly form that opens a MatDialog and helps to populate a textbox in the ngx-formly form
Currently, I have a form containing two fields, name and nickname both of which are textboxes. This form is made out of ngx-formly. Currently on double-clicking the nickname textbox, a MatDialog opens up that contains another form which helps me to populate the nickname textbox on submitting the form in the MatDialog. Instead of double-clicking the nickname textbox, I would like to have a button beside the nickname textbox which says "..." and on clicking the button, it should open the MatDialog, and then it should help me populate the nickname textbox in the parent form.
Sample Stackblitz: https://stackblitz.com/github/m3niza/ngx-formly-modal
I want the double-clicking to be removed and add a button performing the above mentioned action.

My provided code is not working from stackblitz but it will work from local workspace. Github: https://github.com/m3niza/ngx-formly-modal

use https://formly.dev/examples/other/material-prefix-suffix

repo: ngx-formly/ngx-formly


text:
Repository?

Is there a Formly Repository where developers can post their Formly solutions (i.e., custom formatted forms, customize components)?

not currently. feel free to make github gists or similar code blocks, though. You could host them on your own github, tweet about them, etc

authors: MaxKless ccpony


text:
Use localStorage setItem and getItem instead of this.storage[cachePersistenceKey]

I'm submitting a...

[ X ] Bug report
[ ] Feature request
[ ] Documentation issue or request

Current behavior
When I setup the Jest configuration in the generated project, I have an error on the localStorage usage in src/app/core/http/http-cache.service.ts.
Actually, Jest in mocking the localStorageand the http-cache.service.ts is not using the methods setItem and getItem but instead, it is using this.storage[cachePersistenceKey]....
Expected behavior
Use the methods setItem and getItem to communicate with the localStorage
  private saveCacheData() {
    if (this.storage) {
      this.storage.setItem(cachePersistenceKey, JSON.stringify(this.cachedData));
    }
  }

  private loadCacheData() {
    const data = this.storage ? this.storage.getItem(cachePersistenceKey) : null;
    this.cachedData = data ? JSON.parse(data) : {};
  }

Apparently the keyed accessor is not part of the standard (MDN) so we should really make that change.
@jogelin thanks for the report, since you have already identified the fix maybe you want to submit a PR for it yourself ? If for some reason you do not want to, I will do it myself, but you might as well take credit.

authors: bursauxa jogelin

repo: ngx-rocket/generator-ngx-rocket


text:
Best way to translate routes

Hi, thaanks for this perfect library. We are using in our project that lives at production. We are trying to translate routes but I wondered that if there are any practice about that?

Not the best way, but it works https://github.com/gilsdav/ngx-translate-router

Hi I'm facing the same problem did you find any solution ?

This issue opened 3 years ago, i remember i didn't fix.

authors: BojanKogoj besnik26 ozergul

repo: ngx-translate/core


text:
Add ability to set weekend days

Version: v1.12.11
Current Behavior
Current code sets first & last day in the week as weekend days function isWeekend
isWeekend: function(day) {
    return day === 0 || day === 6;
},

Wanted Behavior
It't be great if I can enter custom weekend days(Pass an array of numbers I want to mark as weekend).

I think 'weekend' means Saturday and Sunday.
Why do you need weekend customization?
Do you just want to mark different days of the week as weekends?

In our system, Some countries weekend days are Saturday and Sunday, some other countries have Friday and Saturday as their weekend days and some countries only have Friday.
I could see that there's an option in the calendar to hide weekends, and that depends on isWeekend  function.
I want to mark specific days as weekends, and hide them when narrowWeekend is true

Ah! I undertand.
TOAST UI Calendar 2.0 is expected to support this feature. :)

Great!  Do you know when will v2.0 approx. be released?

@abozanona It will take long time. I'm rewriting this calendar with virtual dom. There are many features those are hard to explain briefly, so check this article.

@abozanona I'll let you know if there is more specific release date.

Would be great to have this feature as the application we are developing is to be used in a country where Friday and Saturday are considered as weekend.

@jungeun-cho @dongsik-yoo
now, are you support this feature or any way to off some days and hide some days?

@luayjal
We don't have any plan for those features.

authors: abozanona adhrinae dongsik-yoo jungeun-cho luayjal pythiantech


text:
IE before 9 is not supported

As of https://github.com/nhn/tui.calendar#-browser-support only IE9+ is supported, so no hacks for IE before 9 are necessary.

Can one of the admins verify this patch?

authors: dilyanpalauzov seonim-ryu


text:
adds a flag for specifying the db name for migrations

Problem
Currently, it seems that the name of the target database for migrations is fixed to "default." This produces the following error when there is no database named "default":
problem applying Hasura metadata: problem adding metadata for the buckets table: status_code: 400\nresponse: {"error":"source with name \"default\" does not exist","path":"$.args","code":"not-exists"}

(see https://github.com/nhost/hasura-storage/issues/184#issue-1972012220)
Solution
To address this issue and provide more flexibility in configuring the target database for migrations, we propose the following solution:

Environment Variable: Introduce a new environment variable named HASURA_DB_NAME to allow users to specify the target database name for migrations.

Configuration Update: Modify the code in hasura-storage/migrations/hasura.go to use the value of HASURA_DB_NAME as the target database name. This ensures that users can configure the target database dynamically.

Fallback to "default": If HASURA_DB_NAME is not set, the code should default to "default" to maintain backward compatibility.

Notes

Users can set the HASURA_DB_NAME environment variable to configure the target database for migrations to their desired database name.

This change enhances the flexibility of the migration process, especially in cases where the database name is not "default."

Reviewers are encouraged to verify that the documentation has been updated to reflect the new configuration option.

With this solution, users can customize the target database for migrations by setting the HASURA_DB_NAME environment variable, which eliminates the error associated with the fixed "default" database name.

Thanks for the PR, there is a minor typo but other than that it looks good.

Thanks!

When can we expect to see this in the Docker image?

Just released.

authors: ChrisSG dbarrosop

repo: nhost/hasura-storage


text:
🛑 Auth is down

In efa829f, Auth ($AUTH_URL) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Auth is back up in 956fc09 after 23 minutes.

repo: nhs-work/test-upptime