text:
This will lock my program and cpu over 100%

This will lock my program and cpu over 100%
jstack
	at co.elastic.apm.agent.bci.ElasticApmAgent.ensureInstrumented(ElasticApmAgent.java:694) 	- waiting to lock <0x00000005c9a7a690> (a java.lang.Class for co.elastic.apm.agent.bci.ElasticApmAgent)
	at co.elastic.apm.agent.bci.ElasticApmAgent.ensureInstrumented(ElasticApmAgent.java:694) 	- waiting to lock <0x00000005c9a7a690> (a java.lang.Class for co.elastic.apm.agent.bci.ElasticApmAgent)
	at co.elastic.apm.agent.bci.ElasticApmAgent.ensureInstrumented(ElasticApmAgent.java:694) 	- waiting to lock <0x00000005c9a7a690> (a java.lang.Class for co.elastic.apm.agent.bci.ElasticApmAgent)
-- waiting to lock * 100+
detail:
co.elastic.apm.agent.bci.ElasticApmAgent#ensureInstrumented

is less line ?

Hi @MJorcen can you provide a bit more details on when this issue appears ?

What is agent configuration ? Are you using trace_methods by any chance ?
Can you provide agent debug logs by adding -Dlog_level=debug to your JVM parameters ?
Can you provide the complete jstack or thread dump output ?

java	-version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2~bpo8+1-b14)
OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
jvm
"-Xms6048m","-Xmn4048m","-Xmx6048m","-XX:SurvivorRatio=10","-XX:-UseAdaptiveSizePolicy","-XX:MetaspaceSize=256M","-XX:-UseAdaptiveSizePolicy","-XX:+HeapDumpOnOutOfMemoryError","-XX:HeapDumpPath=/host_tmp/,"-XX:+PrintGC","-XX:+PrintGCDetails","-XX:+PrintHeapAtGC","-XX:+PrintGCApplicationStoppedTime","-XX:+PrintGCApplicationConcurrentTime","-Xloggc:/gc.log","-Djava.security.egd=file:/dev/./urandom","-jar","/boot.jar","--spring.profiles.active=${PROFILES_ACTIVE}","-javaagent:/elastic-apm-agent.jar"
evn:
ELASTIC_APM_SERVER_URLS: http://elasticsearch-xxx.xxxx:8200
ELASTIC_APM_SECRET_TOKEN: xxxxxx
ELASTIC_APM_SERVICE_NAME: xxxxx
ELASTIC_APM_ENVIRONMENT: prod
ELASTIC_APM_APPLICATION_PACKAGES: com.xxx
ELASTIC_APM_TRANSACTION_IGNORE_URLS: /xxx/,/xxx/
ELASTIC_APM_STACK_TRACE_LIMIT: 0
ELASTIC_APM_CAPTURE_BODY: all
ELASTIC_APM_CAPTURE_HEADERS: false
ELASTIC_APM_TRANSACTION_SAMPLE_RATE: 0.001
ELASTIC_APM_SPAN_FRAMES_MIN_DURATION: 30ms

smiple.log
this is simple log ,

the running thread
"http-nio-10114-exec-26" #219 daemon prio=5 os_prio=0 tid=0x00007f897401b000 nid=0xdf runnable [0x00007f88a9ebe000]
java.lang.Thread.State: RUNNABLE
at co.elastic.apm.agent.shaded.asm.MethodWriter.computeAllFrames(MethodWriter.java:1610)
at co.elastic.apm.agent.shaded.asm.MethodWriter.visitMaxs(MethodWriter.java:1546)
at co.elastic.apm.agent.shaded.asm.MethodVisitor.visitMaxs(MethodVisitor.java:772)
at co.elastic.apm.agent.shaded.asm.MethodVisitor.visitMaxs(MethodVisitor.java:772)
at co.elastic.apm.agent.shaded.bytebuddy.asm.Advice$AdviceVisitor.visitMaxs(Advice.java:9912)
at co.elastic.apm.agent.shaded.asm.ClassReader.readCode(ClassReader.java:2636)
at co.elastic.apm.agent.shaded.asm.ClassReader.readMethod(ClassReader.java:1488)
at co.elastic.apm.agent.shaded.asm.ClassReader.accept(ClassReader.java:718)
at co.elastic.apm.agent.shaded.asm.ClassReader.accept(ClassReader.java:401)
at co.elastic.apm.agent.shaded.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:3827)
at co.elastic.apm.agent.shaded.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2166)
at co.elastic.apm.agent.shaded.bytebuddy.dynamic.scaffold.inline.RedefinitionDynamicTypeBuilder.make(RedefinitionDynamicTypeBuilder.java:224)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:10327)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10263)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1600(AgentBuilder.java:10029)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:10648)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:10595)
at java.security.AccessController.doPrivileged(Native Method)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10186)
at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
at sun.reflect.GeneratedMethodAccessor515.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Dispatcher$ForJava6CapableVm.retransformClasses(AgentBuilder.java:6869)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(AgentBuilder.java:7148)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(AgentBuilder.java:6993)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(AgentBuilder.java:4855)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:9463)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:9384)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$Delegator.installOn(AgentBuilder.java:10986)
at co.elastic.apm.agent.bci.ElasticApmAgent.ensureInstrumented(ElasticApmAgent.java:725)
- locked <0x00000005c9a7a690> (a java.lang.Class for co.elastic.apm.agent.bci.ElasticApmAgent)
at co.elastic.apm.agent.bci.DynamicTransformerImpl.ensureInstrumented(DynamicTransformerImpl.java:36)
at co.elastic.apm.agent.concurrent.JavaConcurrent.captureContext(JavaConcurrent.java:113)
at co.elastic.apm.agent.concurrent.JavaConcurrent.withContext(JavaConcurrent.java:136)
at co.elastic.apm.agent.concurrent.ExecutorInstrumentation$ExecutorCallableInstrumentation.onSubmit(ExecutorInstrumentation.java:156)
at java.lang.invoke.LambdaForm$DMH/285377351.invokeStatic_LL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/1833848849.linkToTargetMethod(LambdaForm$MH)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at com.zaxxer.hikari.pool.HikariPool.addBagItem(HikariPool.java:339)
at com.zaxxer.hikari.util.ConcurrentBag.borrow(ConcurrentBag.java:140)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:180)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:82)
at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:68)
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336)
at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:85)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
at sun.reflect.GeneratedMethodAccessor526.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
at com.sun.proxy.$Proxy168.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:230)
at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:58)
at com.sun.proxy.$Proxy231.select(Unknown Source)
at sun.reflect.GeneratedMethodAccessor666.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy232.select(Unknown Source)
at com.xxxx.someBiz
at com.xxxx.someBiz
at com.xxxx.someBiz
at com.xxxx.someBiz
at com.xxxx.someBiz$$FastClassBySpringCGLIB$$fc4f7b7b.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:100)
at com.xxxx.someBiz
at com.xxxx.someBiz
at sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at com.xxxx.someBiz$$EnhancerBySpringCGLIB$$34334e07.get()
at sun.reflect.GeneratedMethodAccessor641.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.xxxx.commons.filter.RestApiFilter.safeFilter(RestApiFilter.java:62)
at com.xxxx.commons.filter.RestApiFilter.doFilter(RestApiFilter.java:49)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000005cfbc08d0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000005d06db6e0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Can I do this .
I try to fix it and wait for the problem to recur .

I try to fix it and add adding -Dlog_level=debug  .
Now it's waiting to happen again
If I change to this, what bad things will happen ?

Hi @MJorcen ,
The global lock on ElasticApmAgent.class is here to ensure that there is at most a single concurrent execution of this method.
Relying on the class that is being instrumented (classToInstrument) is not enough as it might trigger classloading and thus nested calls to ensureInstrumented. While it could appear to reduce your issue in this case, we would not be able to apply this generally.
Just to be sure, which version of the agent are you using here ? Please note that we recently changed our OSS header, thus there might be a +5/-5 line offset between the lines in the stack dump and the current state of master branch.
Also, as of version 1.23.0 we avoid re-computing stack frames for Java 7+ code (see changelog)
One possible hypothesis here is that the instrumentation of a single class within the JDK by the concurent plugin is taking way more time than usual.
From this part in the stack:
at co.elastic.apm.agent.shaded.asm.MethodWriter.computeAllFrames(MethodWriter.java:1610)
...
at co.elastic.apm.agent.bci.ElasticApmAgent.ensureInstrumented(ElasticApmAgent.java:725)
- locked <0x00000005c9a7a690> (a java.lang.Class for co.elastic.apm.agent.bci.ElasticApmAgent)
at co.elastic.apm.agent.bci.DynamicTransformerImpl.ensureInstrumented(DynamicTransformerImpl.java:36)
at co.elastic.apm.agent.concurrent.JavaConcurrent.captureContext(JavaConcurrent.java:113)
at co.elastic.apm.agent.concurrent.JavaConcurrent.withContext(JavaConcurrent.java:136)
at co.elastic.apm.agent.concurrent.ExecutorInstrumentation$ExecutorCallableInstrumentation.onSubmit(ExecutorInstrumentation.java:156)
at java.lang.invoke.LambdaForm$DMH/285377351.invokeStatic_LL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/1833848849.linkToTargetMethod(LambdaForm$MH)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at com.zaxxer.hikari.pool.HikariPool.addBagItem(HikariPool.java:339)
...

We see that :

HikariCP delegates task execution to a standard ExecutorService
our instrumentation calls ensureInstrumented on the task to propagate context
other parts upper in the stack are related to byte-buddy/ASM recomputing frames, which is likely the most expensive part of instrumentation.

Thus here, it seems to me that it could be something that was fixed with 1.23.0 as the class that is being instrumented is within the JDK and should already be compiled to Java 8 bytecode, thus no stack-frame re-computation is required.
Could you just confirm which version of the agent you are using here ? If it's not 1.24.0, then try updating first.
Also, how frequently does this issue happens ? Does it happen on every app start ? 50% of the time or less ?

Yes, most of this is when it starts and It happened in version 1.18.0
When I changed the code on 1.24.0, it didn't appear again. Maybe you have fixed it. Thank you.
You are great 😘

Thanks a lot for your feedback @MJorcen ! That definitely helps as those issues are quite hard to reproduce in the field.

authors: MJorcen SylvainJuge

repo: elastic/apm-agent-java


text:
Replace apache download tasks with custom ivy repository.

Hadoop ecosystem tarballs are hosted on Apache's download mirrors only instead of in maven repositories. Previously we were downloading these artifacts from a round robin of mirror endpoints, using securely obtained hashes to validate the downloaded files even from non HTTPS sources.
This PR removes those download tasks, replacing it with a custom Ivy repository backed by an HTTPS enabled apache mirror. This allows us to ensure the artifacts are securely downloaded and cached by Gradle instead of relying on task outputs like the old mirror download task.

One thing to keep in mind here is that Gradle doesn't do artifact checksum verification by default. In fact, this capability was only recently added in Gradle 6.2. As it is, this PR removes the checksum validation for these artifacts. If we want to retain that capability, we'll need to enabled checksum verification in Gradle as described here.

The dependency verification features look really useful, but I think they might be a bit out of scope. It looks like we mostly want the verification for downloads from apache mirrors, but it's a global setting that will require upkeep outside of normal project configuration. Additionally, we would need to maintain a number of unverified configurations since we rely on the local file repo in snapshot and release builds.

source: gharchive/pull-request

authors: jbaiera mark-vieira

repo: elastic/elasticsearch-hadoop

license_type: permissive

license_source: bigquery


text:
Exception is thrown when try to deserialize an object using NTS

NEST/Elasticsearch.Net version: 2.4.5
Elasticsearch version: 2.3.5
Description of the problem including expected versus actual behavior:
Expected: Query for shape should return valid results.
Actual: Query throws exception.
Steps to reproduce:

I'll write the relevant code below, hoping it will be enough.

I'm using NEST with NTS in order to be able to query OSM data, more specifically all the highways in an area.
So I needed to write a JsonNetSerializer to incorporate the Json converters from NTS (Net Topology Suite):
public class GeoJsonSerializer : JsonNetSerializer
    {
        public GeoJsonSerializer(IConnectionSettingsValues settings) : base(settings)
        {
            OverwriteDefaultSerializers((s, cvs) =>
            {
                s.Converters.Add(new CoordinateConverter());
                s.Converters.Add(new GeometryConverter());
                s.Converters.Add(new FeatureCollectionConverter());
                s.Converters.Add(new FeatureConverter());
                s.Converters.Add(new AttributesTableConverter());
                s.Converters.Add(new ICRSObjectConverter());
                s.Converters.Add(new GeometryArrayConverter());
                s.Converters.Add(new EnvelopeConverter());
            });
        }

        public GeoJsonSerializer(IConnectionSettingsValues settings, JsonConverter statefulConverter) : base(settings, statefulConverter)
        {
            OverwriteDefaultSerializers((s, cvs) =>
            {
                s.Converters.Add(new CoordinateConverter());
                s.Converters.Add(new GeometryConverter());
                s.Converters.Add(new FeatureCollectionConverter());
                s.Converters.Add(new FeatureConverter());
                s.Converters.Add(new AttributesTableConverter());
                s.Converters.Add(new ICRSObjectConverter());
                s.Converters.Add(new GeometryArrayConverter());
                s.Converters.Add(new EnvelopeConverter());
            });
        }
    }

I then indexed GeoJSON.Features.Feature which worked well.
The last part is what failed - query the data back using the following code:
        public async Task<List<Feature>> GetHighways(LatLng northEast, LatLng southWest)
        {
            var response = await _elasticClient.SearchAsync<Feature>(
                s => s.Index(OSM_HIGHWAYS_INDEX).Size(5000).Query(
                    q => q.GeoShapeEnvelope(
                        e => e.Coordinates(new List<GeoCoordinate>
                            {
                                new GeoCoordinate(northEast.lat, northEast.lng),
                                new GeoCoordinate(southWest.lat, southWest.lng)
                            }).Field(f => f.Geometry)
                            .Relation(GeoShapeRelation.Intersects)
                    )
                )
            );
            return response.Documents.ToList();
        }

this throw the following exception:
Elasticsearch.Net.UnexpectedElasticsearchClientException: Unexpected token when deserializing object:
StartObject. Path 'hits.hits[1]', line 41, position 8. --->
Newtonsoft.Json.JsonSerializationException: Unexpected token when deserializing object:
StartObject. Path 'hits.hits[1]', line 41, position 8.

I looked at the json results using the web interface with the following query and everything looked OK:
{
    "query":{
        "bool": {
            "must": {
                "match_all": {}
            },
            "filter": {
                "geo_shape": {
                    "geometry": {
                        "shape": {
                            "type": "envelope",
                            "coordinates" : [[35.0516, 31.7553], [35.0251, 31.7467]]
                        }
                    }
                }
            }
        }
    }
}

I also tried my luck with the following issues:  #1849, #2282 but didn't manage to make it work.
As a side note, I have a workaround using object without the GeoJsonSerializer, but it feels hacky as hell...

@HarelM Can you put together a small reproducible example in something like Linqpad? Take a look at https://github.com/elastic/elasticsearch-net/issues/2445#issuecomment-264603001 for an example Linqpad program that will get you 90% of the way there.
My initial suspicion is that the GEOJSON Feature is not indexed in the form you expect it to be; Feature and FeatureCollection are not supported geo_shape types.
It sounds like deserialization is failing into your POCO that contains the Feature, but I think this will be more clear with an example.

I'll post the code here, it's relatively small, I use VS:
You'll need the following nuget packages (package.json):
<packages>
  <package id="Elasticsearch.Net" version="2.5.0" targetFramework="net452" />
  <package id="GeoAPI" version="1.7.4" targetFramework="net452" />
  <package id="NEST" version="2.5.0" targetFramework="net452" />
  <package id="NetTopologySuite" version="1.14" targetFramework="net452" />
  <package id="NetTopologySuite.IO.GeoJSON" version="1.14" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="9.0.1" targetFramework="net452" />
</packages>

code:
using System;
using System.Collections.Generic;
using System.Linq;
using Elasticsearch.Net;
using GeoAPI.Geometries;
using Nest;
using NetTopologySuite.Features;
using NetTopologySuite.Geometries;
using NetTopologySuite.IO;
using NetTopologySuite.IO.Converters;
using Newtonsoft.Json;
using Feature = NetTopologySuite.Features.Feature;

namespace NESTBug
{
    public class GeoJsonSerializer : JsonNetSerializer
    {
        public GeoJsonSerializer(IConnectionSettingsValues settings) : base(settings)
        {
            OverwriteDefaultSerializers((s, cvs) =>
            {
                s.Converters.Add(new CoordinateConverter());
                s.Converters.Add(new GeometryConverter());
                s.Converters.Add(new FeatureCollectionConverter());
                s.Converters.Add(new FeatureConverter());
                s.Converters.Add(new AttributesTableConverter());
                s.Converters.Add(new ICRSObjectConverter());
                s.Converters.Add(new GeometryArrayConverter());
                s.Converters.Add(new EnvelopeConverter());
            });
        }

        public GeoJsonSerializer(IConnectionSettingsValues settings, JsonConverter statefulConverter) : base(settings, statefulConverter)
        {
            OverwriteDefaultSerializers((s, cvs) =>
            {
                s.Converters.Add(new CoordinateConverter());
                s.Converters.Add(new GeometryConverter());
                s.Converters.Add(new FeatureCollectionConverter());
                s.Converters.Add(new FeatureConverter());
                s.Converters.Add(new AttributesTableConverter());
                s.Converters.Add(new ICRSObjectConverter());
                s.Converters.Add(new GeometryArrayConverter());
                s.Converters.Add(new EnvelopeConverter());
            });
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            const string NEST_BUG_INDEX = "nestbug";
            var client = Init(NEST_BUG_INDEX);
            UpdateData(client, new List<Feature>
            {
                new Feature(new LineString(new []
                {
                    new Coordinate(1,1),
                    new Coordinate(2,2),
                    new Coordinate(10.5, 10.5),
                    new Coordinate(11, 11),
                }), new AttributesTable()),
                new Feature(new LineString(new []
                {
                    new Coordinate(10.5, 10.5),
                    new Coordinate(10.8, 10.8),
                }),new AttributesTable()),
                new Feature(new LineString(new []
                {
                    new Coordinate(12, 12),
                    new Coordinate(13, 13),
                }),new AttributesTable()),
            }, NEST_BUG_INDEX);
            var highways = GetHighways(client, NEST_BUG_INDEX);
            Console.WriteLine(highways.Count);
        }

        private static ElasticClient Init(string NEST_BUG_INDEX)
        {
            var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200/"));
            var connectionString = new ConnectionSettings(
                    pool,
                    new HttpConnection(),
                    new SerializerFactory(s => new GeoJsonSerializer(s)))
                .DefaultIndex(NEST_BUG_INDEX)
                .PrettyJson();
            var elasticClient = new ElasticClient(connectionString);
            if (elasticClient.IndexExists(NEST_BUG_INDEX).Exists)
            {
                elasticClient.DeleteIndex(NEST_BUG_INDEX);
            }
            elasticClient.CreateIndex(NEST_BUG_INDEX,
                    c => c.Mappings(
                        ms => ms.Map<Feature>(m =>
                            m.Properties(ps => ps.GeoShape(g => g.Name(f => f.Geometry)
                                .Tree(GeoTree.Geohash)
                                .TreeLevels(10)
                                .DistanceErrorPercentage(0.2))))));
            return elasticClient;
        }

        public static void UpdateData(ElasticClient client, List<Feature> features, string index)
        {
            var writer = new GeoJsonWriter();
            var id = 0;
            client.Bulk(bulk =>
            {
                foreach (var feature in features)
                {
                    bulk.Index<Feature>(i => i.Index(index).Document(feature).Id(id++));
                }
                return bulk;
            });
        }

        public static List<Feature> GetHighways(ElasticClient client, string index)
        {
            var response = client.Search<Feature>(
                s => s.Index(index).Size(5000).Query(
                    q => q.GeoShapeEnvelope(
                        e => e.Coordinates(new List<GeoCoordinate>
                            {
                                new GeoCoordinate(10, 10),
                                new GeoCoordinate(11, 11)
                            }).Field(f => f.Geometry)
                            .Relation(GeoShapeRelation.Intersects)
                    )
                )
            );
            return response.Documents.ToList();
        }
    }
}

@HarelM I've spent some time looking at this.
It's a bug in FeatureConverter, which has been fixed in https://github.com/NetTopologySuite/NetTopologySuite/commit/70b6755ec84992a4b72ceec3a30b1459068af85b but is not in the latest released NetTopologySuite.IO.GeoJSON 1.14 on nuget.
In FeatureConverter in NetTopologySuite.IO.GeoJSON 1.14 (there's no tag in the repo for 1.14, but this looks to be the last commit), when it gets to the EndObject token for the Feature, it reads to the next token before returning the deserialized Feature. The reader is now at the EndObject token of the IHit<T>.
At this point, Json.NET's  JsonSerializerInternalReader.PopulateObject() method, which is deserializing the outer HIt<T>, reads to the next token again in the predicate of the do / while(), since it has just finished deserializing the _source T property, Feature in this case. In doing this, the reader advances to the StartObject token of the next IHit<T> in the hits array, which throws the exception you see.
The fix for now would be to patch a converter for Feature and use that, at least until the next version of NetTopologySuite is out on Nuget. Here's a version that works, along with a complete example
public static void Main()
{
	const string NEST_BUG_INDEX = "nestbug";

	var settings = new ConnectionSettings(
		new SingleNodeConnectionPool(new Uri("http://localhost:9200")),
		new HttpConnection(),
		new SerializerFactory(c => new GeoJsonSerializer(c)))
		.DefaultIndex(NEST_BUG_INDEX);

	var client = new ElasticClient(settings);

	client.CreateIndex(NEST_BUG_INDEX, c => c
		.Mappings(ms => ms
			.Map<Feature>(m => m
				.Properties(ps => ps
					.GeoShape(g => g
							.Name(f => f.Geometry)
							.Tree(GeoTree.Geohash)
							.TreeLevels(10)
							.DistanceErrorPercentage(0.2)
					)
				)
			)
		)
	);

	var features = new List<Feature>
	{
		new Feature(new LineString(new[]
		{
			new Coordinate(1, 1),
			new Coordinate(2, 2),
			new Coordinate(10.5, 10.5),
			new Coordinate(11, 11),
		}), new AttributesTable()),
		new Feature(new LineString(new[]
		{
			new Coordinate(10.5, 10.5),
			new Coordinate(10.8, 10.8),
		}), new AttributesTable()),
		new Feature(new LineString(new[]
		{
			new Coordinate(12, 12),
			new Coordinate(13, 13),
		}), new AttributesTable()),
	};

	var id = 0;
	client.Bulk(bulk => bulk.IndexMany(features, (i, f) => i.Id(id++)));

	client.Refresh(NEST_BUG_INDEX);

	var response = client.Search<Feature>(s => s
		.Size(5000)
		.Query(q => q
			.GeoShapeEnvelope(e => e
				.Field(f => f.Geometry)
				.Coordinates(new List<GeoCoordinate>
				{
					new GeoCoordinate(10, 10),
					new GeoCoordinate(11, 11)
				})
				.Relation(GeoShapeRelation.Intersects)
			)
		)
	);
}

public class GeoJsonSerializer : JsonNetSerializer
{
	private static readonly JsonConverter[] NetTopologyConverters = {
		new CoordinateConverter(),
		new GeometryConverter(),
		new FeatureCollectionConverter(),
		new FixedFeatureConverter(),
		new AttributesTableConverter(),
		new ICRSObjectConverter(),
		new GeometryArrayConverter(),
		new EnvelopeConverter()
	};

	protected override IList<Func<Type, JsonConverter>> ContractConverters =>
		new List<Func<Type, JsonConverter>>
		{
			t => NetTopologyConverters.FirstOrDefault(c => c.CanConvert(t))
		};

	public GeoJsonSerializer(IConnectionSettingsValues settings) : base(settings)
	{
	}
}

/// Converts Feature object to its JSON representation.
public class FixedFeatureConverter : JsonConverter
{
    /// Writes the JSON representation of the object.
    /// <param name="writer">The <see cref="T:Newtonsoft.Json.JsonWriter" /> to write to.</param>
    /// <param name="value">The value.</param>
    /// <param name="serializer">The calling serializer.</param>
    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        if (writer == null) throw new ArgumentNullException(nameof(writer));
        if (serializer == null) throw new ArgumentNullException(nameof(serializer));

        var feature = value as IFeature;
        if (feature == null) return;

        writer.WriteStartObject();
        writer.WritePropertyName("type");
        writer.WriteValue("Feature");
        writer.WritePropertyName("geometry");
        serializer.Serialize(writer, feature.Geometry);
        writer.WritePropertyName("properties");
        serializer.Serialize(writer, feature.Attributes);
        writer.WriteEndObject();
    }

    /// Reads the JSON representation of the object.
    /// <param name="reader">The <see cref="T:Newtonsoft.Json.JsonReader" /> to read from.</param>
    /// <param name="objectType">Type of the object.</param>
    /// <param name="existingValue">The existing value of object being read.</param>
    /// <param name="serializer">The calling serializer.</param>
    /// <returns>The object value.</returns>
    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        reader.Read();
        if (reader.TokenType != JsonToken.PropertyName || (string)reader.Value != "type")
            throw new ArgumentException("Expected token 'type' not found.");
        reader.Read();
        if (reader.TokenType != JsonToken.String && (string)reader.Value != "Feature")
            throw new ArgumentException("Expected value 'Feature' not found.");
        reader.Read();
        object id = null;
        var feature = new Feature();

        while (reader.TokenType == JsonToken.PropertyName)
        {
            switch ((string)reader.Value)
            {
                case "id":
                    reader.Read();
                    id = reader.Value;
                    reader.Read();
                    continue;
                case "bbox":
                    serializer.Deserialize<Envelope>(reader);
                    continue;
                case "geometry":
                    reader.Read();
                    if (reader.TokenType == JsonToken.Null)
                    {
                        reader.Read();
                        continue;
                    }
                    if (reader.TokenType != JsonToken.StartObject)
                        throw new ArgumentException("Expected token '{' not found.");

                    feature.Geometry = serializer.Deserialize<IGeometry>(reader);

                    if (reader.TokenType != JsonToken.EndObject)
                        throw new ArgumentException("Expected token '}' not found.");

                    reader.Read();
                    continue;
                case "properties":
                    reader.Read();
                    if (reader.TokenType != JsonToken.StartObject)
                        throw new ArgumentException("Expected token '{' not found.");

                    feature.Attributes = serializer.Deserialize<AttributesTable>(reader);

                    if (reader.TokenType != JsonToken.EndObject)
                        throw new ArgumentException("Expected token '}' not found.");
                    reader.Read();
                    continue;
                default:
                    reader.Read();
                    while (reader.TokenType != JsonToken.PropertyName)
                        reader.Read();
                    continue;
            }
        }
        if (reader.TokenType != JsonToken.EndObject)
            throw new ArgumentException("Expected token '}' not found.");

        if (feature.Attributes != null && id != null && !feature.Attributes.Exists("id"))
            feature.Attributes.AddAttribute("id", id);

        return feature;
    }

    ///
    /// Determines whether this instance can convert the specified object type.
    ///
    /// <param name="objectType">Type of the object.</param>
    /// <returns>
    ///   <c>true</c> if this instance can convert the specified object type; otherwise, <c>false</c>.
    /// </returns>
    public override bool CanConvert(Type objectType)
    {
        return typeof(Feature).IsAssignableFrom(objectType);
    }
}

Thank you very much for your detailed response and investigation!
I'll wait for the NTS version release for this fix assuming it is fixed as you say.
Current workaround with object can be seen here - which is a hell lot simpler, although less type-safe :-)
https://github.com/IsraelHikingMap/Site/blob/master/IsraelHiking.DataAccess/ElasticSearch/ElasticSearchGateway.cs
I'm guessing the performance impact is relatively small so I don't think I need to worry about it.
Thanks again!

No worries. the converter for features above works (I've tested it), so you could avoid returning object and using GeoJsonReader by using it instead, up to you 😃

source: gharchive/issue

authors: HarelM russcam

repo: elastic/elasticsearch-net

license_source: github-api


text:
Validate API action

Work in progress...

You have updated some API definitions, great work!
Following you can find the validation results for the APIs you have changed.
bulk

```sh
- Validating endpoints
✔ bulk request has been successfully validated!
✔ 90 out of 90 test request cases are passing.
✔ 108 out of 108 test response cases are passing.
</summary>

`async_search.delete`
<summary>
```sh
- Validating endpoints
✔ async_search.delete request has been successfully validated!

✔ 1 out of 1 test request cases are passing.
✔ 1 out of 1 test response cases are passing.

eql.delete

```sh
- Validating endpoints
✔ eql.delete request has been successfully validated!
✔ 0 out of 0 test request cases are passing.
✔ 0 out of 0 test response cases are passing.
</summary>

`eql.get`
<summary>
```sh
- Validating endpoints
✔ eql.get request has been successfully validated!

✔ 0 out of 0 test request cases are passing.
✔ 0 out of 0 test response cases are passing.

eql.get_status

```sh
- Validating endpoints
✔ eql.get_status request has been successfully validated!
✔ 0 out of 0 test request cases are passing.
✔ 0 out of 0 test response cases are passing.
</summary>

`eql.search`
<summary>
```sh
- Validating endpoints
✔ eql.search request has been successfully validated!

✔ 0 out of 0 test request cases are passing.
✔ 0 out of 0 test response cases are passing.

You have updated some API definitions, great work!
Following you can find the validation results for the APIs you have changed.

`bulk`
- Validating endpoints
✔ bulk request has been successfully validated!

✔ 90 out of 90 test request cases are passing.
✔ 108 out of 108 test response cases are passing.

`async_search.delete`
- Validating endpoints
✔ async_search.delete request has been successfully validated!

✔ 1 out of 1 test request cases are passing.
✔ 1 out of 1 test response cases are passing.

`eql.delete`
- Validating endpoints
✔ eql.delete request has been successfully validated!

✔ 0 out of 0 test request cases are passing.
✔ 0 out of 0 test response cases are passing.

`eql.get`
- Validating endpoints
✔ eql.get request has been successfully validated!

✔ 0 out of 0 test request cases are passing.
✔ 0 out of 0 test response cases are passing.

`eql.get_status`
- Validating endpoints
✔ eql.get_status request has been successfully validated!

✔ 0 out of 0 test request cases are passing.
✔ 0 out of 0 test response cases are passing.

`eql.search`
- Validating endpoints
✔ eql.search request has been successfully validated!

✔ 0 out of 0 test request cases are passing.
✔ 0 out of 0 test response cases are passing.

repo: elastic/elasticsearch-specification


text:
[logstash] disable privileged container in psp

This commit disable privilged container when enabling PodSecurityPolicy
as Logstash container doesn't use any host device.

This looks like it might be a legitimate test failure https://devops-ci.elastic.co/job/elastic+helm-charts+pull-request+template-testing/1123/CHART=logstash,label=docker&&virtual/console

This looks like it might be a legitimate test failure https://devops-ci.elastic.co/job/elastic+helm-charts+pull-request+template-testing/1123/CHART=logstash,label=docker&&virtual/console

This looks like it might be a legitimate test failure devops-ci.elastic.co/job/elastic+helm-charts+pull-request+template-testing/1123/CHART=logstash,label=docker&&virtual/console

Fixed @Conky5 please re-review

This looks like it might be a legitimate test failure devops-ci.elastic.co/job/elastic+helm-charts+pull-request+template-testing/1123/CHART=logstash,label=docker&&virtual/console

Fixed @Conky5 please re-review

repo: elastic/helm-charts


text:
[elasticsearch] Make ingress hosts optional

This PR allows the ingress hosts definition to be made empty, which is useful for POC situations where we really don't care about hostnames, for example in GKE using an internal loadbalancer with container native loadbalancing:
service:
  annotations:
    cloud.google.com/neg: '{"ingress": true}'

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: "gce-internal"
  path: /
  hosts:

Without this PR the chart would generate an ingress without rules given above config, which is not accepted by kubernetes.
With this PR it generates a proper ingress template without a hostname tied to it.

[ x] Chart version not bumped (the versions are all bumped and released at the same time)
[ x] README.md updated with any new values or changes
[x ] Updated template tests in ${CHART}/tests/*.py
[ ] Updated integration tests in ${CHART}/examples/*/test/goss.yaml

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

Hi @wouterdegeus,
Thanks for opening this PR.
We'll try to review it soon. Please be patient.

Hi @wouterdegeus,
Thanks for opening this PR.
We'll try to review it soon. Please be patient.

Related to #536

Related to #536

jenkins test this please

Hi @wouterdegeus,
Since #994, you can create an ingress with empty hosts using the following values:
ingress:
  enabled: true
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
  hosts:
    - host: ''
      paths:
        - path: /

I tested it and had exactly the same ingress generated as using your PR.
Can you confirm this is working for your use case?

Just tested, this works indeed!

Just tested, this works indeed!

Awesome 👍
Closing your PR then.

authors: elasticmachine jmlrt wouterdegeus


text:
use same env variable as application

Signed-off-by: Riccardo Piccoli riccardo.piccoli@softonic.com

[x] Chart version not bumped (the versions are all bumped and released at the same time)
[ ] README.md updated with any new values or changes
[ ] Updated template tests in ${CHART}/tests/*.py
[ ] Updated integration tests in ${CHART}/examples/*/test/goss.yaml

It appears that env vars used by kibana are ELASTICSEARCH_USERNAME and ELASTICSEARCH_PASSWORD, but readiness uses ELASTIC_USERNAME and ELASTIC_PASSWORD. This causes kibana to be unready, unless we also configure ELASTIC_ vars.

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

jenkins test this please

jenkins test this please

jenkins test this please

authors: Crazybus elasticmachine rccrdpccl


text:
Investigate robustness of throughput throttling

A user has reported odd throughput throttling numbers, e.g. instead of a targeted 100 ops/s they are seeing 105 ops/s. While it is not possible to exactly hit the target throughput (e.g. Rally might hit 200.1 ops/s instead of 200.0 ops/s) this variation is too much.
This could be caused by an inaccurate system timer and we should investigate whether this is reproducible.
More environment info:

OS (uname -a): Linux IP 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux (Ubuntu 14.04)
Python (python3 -c "import sys ; print(sys.implementation)"): namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-34', hexversion=50594800, name='cpython', version=sys.version_info(major=3, minor=4, micro=3, releaselevel='final', serial=0))
Hardware: AWS EC2, t2.large instance type

Closing for now. We will reopen if this comes up again.

repo: elastic/rally


text:
Automatic retry on networking issues

Fixes #10
2.0.4 (November 10, 2020)

Bump dependencies
Automatically & immediately retry 5 times on network failure
All network failures trigger rebounds when the enable rebound option is set

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

tested in flow node-> rest api v2 with rest api requests to https://cdn.elastic.io/test.json
~ 285400 requests were done with no network-related errors thrown
so it looks like issue is fixed as here: https://github.com/elasticio/elasticio/issues/4874#issue-739936718 was mentioned 210 000 msgs with 213 network-related errors

authors: CLAassistant jhorbulyk stas-fomenko


text:
fix: return Connection established instead OK

This commit changes to return 200 Connection established instead 200 OK.
   Example of a response:

         HTTP/1.0 200 Connection established
         Proxy-agent: Netscape-Proxy/1.1

         ...data tunnelled from the server...

ref: https://datatracker.ietf.org/doc/html/draft-luotonen-web-proxy-tunneling-01#section-3.2

Thanks!

authors: elazarl hgsgtk

repo: elazarl/goproxy


text:
Missing styles with @media nesting (conflict with postcss-nested?)

So the problem:
Initial code:
.b-product-single {

  &__main {
    @media screen and (min-width: 991px) {
      display: grid;
      grid-template-columns: 1fr 250px;
      grid-column-gap: var(--large-margin);
    }
  }

  &__image-wrapper {
    text-align: right;
    overflow: hidden;

    img {
      max-width: 100%;
      height: auto;
      object-fit: scale-down;
    }
  }
}

results to by a postcss-nested plugin:
 @media screen and (min-width: 991px) {

  .b-product-single__main {
      display: grid;
      grid-template-columns: 1fr 250px;
      grid-column-gap: var(--large-margin)
  }

  .b-product-single__image-wrapper {
    text-align: right;
    overflow: hidden;
  }

  .b-product-single__image-wrapper img {
      max-width: 100%;
      height: auto;
      object-fit: scale-down;
    }
    }

Where .b-product-single__main just disappears after postcss-rtlcss.
If I replace initial code with stright result of postcss-nested I cannot see the problem.
My env:
node -v
v16.14.0

package.json:
    "autoprefixer": "^10.4.2",
    "cssnano": "^5.0.17",
    "postcss": "^8.4.6",
    "postcss-nested": "^5.0.6",
    "postcss-rtlcss": "^3.5.2",

    ...
   "postcss-loader": "^6.2.1",
   "webpack": "^5.65.0",
   "webpack-cli": "^4.9.2",

postcss.config.js:
const postcssRTLCSS = require('postcss-rtlcss')
const cssnano = require('cssnano')
const postcssNested = require('postcss-nested')

module.exports = (ctx) => {
  const isProduction = ctx.env === 'production'
  return {
    plugins: [
      postcssNested(),
      postcssRTLCSS({ mode: 'override'}),
      require('autoprefixer'),
      isProduction && cssnano(({
        preset: 'default',
      })),
    ]
  }
}

If I rollback to postcss-rtlcss@2 it works. So may be version 3.x has some conflicts with other postcss plugins?
And another way to make it work:
.b-product-single {

  &__main {
    /* Is to write any rule */
   display: block;
    @media screen and (min-width: 991px) {
      display: grid;
      grid-template-columns: 1fr 250px;
      grid-column-gap: var(--large-margin);
    }
  }

In this way the code results to:
.b-product-single__main {
    display: block;
  }

  @media screen and (min-width: 991px) {

  .b-product-single__main {
      display: grid;
      grid-template-columns: 1fr 250px;
      grid-column-gap: var(--large-margin)
  }
    }

And a last way to make it work is to remove all trailing left-right specific rules:
&__main {
    @media screen and (min-width: 991px) {
      display: grid;
      grid-template-columns: 1fr 250px;
      grid-column-gap: var(--large-margin);
    }
  }

  &__image-wrapper {
    /* We can see a text-align: right; here, so remove it and see what happens: */
    /* text-align: right; */
    overflow: hidden;

    img {
      max-width: 100%;
      height: auto;
      object-fit: scale-down;
    }
  }

Result:
@media screen and (min-width: 991px) {

  .b-product-single__main {
      display: grid;
      grid-template-columns: 1fr 250px;
      grid-column-gap: var(--large-margin)
  }
    }

  .b-product-single__image-wrapper {
    /* text-align: right; */
    overflow: hidden;
  }

  .b-product-single__image-wrapper img {
      max-width: 100%;
      height: auto;
      object-fit: scale-down;
    }

Well .b-product-single__main doesn't disappears now. Magic.
There is no errors due to a build process. I tested it with webpack and rollup, so loaders and bundlers doesn't matters.

Could you create a small repo with a minimum reproducible example so I can look at the issue closer?

Yeah, I'll try as soon as possible but can't promise it will soon

Perfect @LFFATE, when you have time, no pressure. It is just that it would be easier for me to check what is happening and fix it if I can reproduce it easily.
Regards and thanks.

@elchininet
Hello again. I created a repro repo https://github.com/LFFATE/postcss-rtlcss-issue-122-repro
To start it follow these steps:

clone repo
yarn
yarn parcel src/index.html

css located at src/style.css
postcss config located at postcss.config.js
compiled css will be created at dist/index.*.css

Hi @LFFATE, version 3.5.3 solves this issue.
Regards and thanks for all the help.

@elchininet Great! Thank you!

authors: LFFATE elchininet

repo: elchininet/postcss-rtlcss


text:
please follow redirect

Hi, I'd like to download electron from a location on our corporate sonatype nexus.
The uri returned by nexus gives a redirect (302). Redirect is not followed. package is therefore not downloaded even if it exists.
I get the following error:
npm i electron@1.4.15 --save-dev --reg
http://internalrepository.addr.corp:8081/nexus/content/groups/build.milestones.npm/
(node:11289) fs: re-evaluating native module sources is not supported.
If you are using the graceful-fs module, please update it to a more
recent version.

> electron@1.4.15 postinstall /home/me/wing/node_modules/electron
> node install.js

/home/me/wing/node_modules/electron/install.js:46
  throw err
  ^

Error: Failed to find Electron v1.4.15 for linux-x64 at
http://internalrepository.addr.corp:8081/nexus/service/local/repositories/3rd-party.releases.manual-uploads.hosted/content/io/atom/electron/download/electron/electron/1.4.15/electron-1.4.15-linux-x64.zip
    at Request.<anonymous>
(/home/I301937/wing/node_modules/nugget/index.js:169:61)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at Request.onRequestResponse
(/home/I301937/wing/node_modules/request/request.js:1074:10)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at HTTPParser.parserOnIncomingClient [as onIncoming]
(_http_client.js:474:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23)
    at Socket.socketOnData (_http_client.js:363:20)
    at emitOne (events.js:96:13)

However _http://internalrepository.addr.corp:8081/nexus/service/local/repositories/3rd-party.releases.manual-uploads.hosted/content/io/atom/electron/download/electron/electron/1.4.15/electron-1.4.15-linux-x64.zip_ does exist.
FYI: I use the following env variables to specify a different electron uri:
ELECTRON_CUSTOM_FILENAME="electron-1.4.15-linux-x64.zip"
ELECTRON_MIRROR="http://internalrepository.addr.corp:8081/nexus/service/local/repositories/3rd-party.releases.manual-uploads.hosted/content/io/atom/electron/download/electron/electron/1.4.15"

next is changing how downloading works and now uses got which follows redirects

authors: MarshallOfSound Zougi

repo: electron-userland/electron-download

license: MIT


text:
webpack: MAIN_WINDOW_PRELOAD_WEBPACK_ENTRY is not defined

[x] I have read the contribution documentation for this project.
[x] I agree to follow the code of conduct that this project follows, as appropriate.
[x] I have searched the issue tracker for an issue that matches the one I want to file, without success.

Please describe your issue:
I'm using the repo provided by @tgds found here. While it is in typescript rather than JS, it works no problem using the MAIN_WINDOW_WEBPACK_ENTRY variable.
The documentation for the webpack plugin says

For an entry point with the name main_window two variables will be defined MAIN_WINDOW_WEBPACK_ENTRY and MAIN_WINDOW_PRELOAD_WEBPACK_ENTRY

with this example:
const mainWindow = new BrowserWindow({
  webPreferences: {
    preload: MAIN_WINDOW_PRELOAD_WEBPACK_ENTRY,
  }
});

mainWindow.loadUrl(MAIN_WINDOW_WEBPACK_ENTRY);

However, if I modify the code in the repo provided to match, I get an error:

@G-Rath MAIN_WINDOW_PRELOAD_WEBPACK_ENTRY will only be defined if a preload is defined in your @electron-forge/plugin-webpack config.
Their is no preload configured here --> https://github.com/tgds/electrong-forge-webpack-typescript/blob/master/package.json#L42-L46 so that variable is not set.  I'll update the docs now

@MarshallOfSound awesome, thanks for such a quick response!
It wasn't blocking me from development, but since I wasn't preloading, the script would get loaded over localhost, and electron would give me a couple of security warnings about loading remote content.
Thanks again!

@MarshallOfSound would you mind weighting in on a problem I'm having relating to this?
I've got the preload setup & working like so:
"preload": {
  "js": "./src/renderer/index.ts"
}

However, that results in the code being run twice, as it's also injected into my html file via a script tag:
<script type="text/javascript" src="../main_window/index.js"></script>
I'm confused over what's actually meant to be happening - I can't find anything in the docs about @electron-forge/plugin-webpack injecting a link to the js, but it seemed to be how the whole setup worked.
If I remove the "js": "./src/renderer/index.ts", entry point, webpack throws an error, and if I don't use preload, I get warnings from electron about having requesting a remote resource with nodeIntegration: true 😕
I feel like I'm missing a single piece in the puzzle, and would be really grateful if you could point me in the right direction on how to handle this behaviour :)

@G-Rath Two things.

During development it will complain about a "remote" server because it loads your app from localhost via http.  This is OK, it doesn't complain once packaged as it doesn't use localhost in prod

If you want to use a preload it should be a different file, E>g. src/renderer/preload.ts.

What worked for me was:
"entryPoints": [
                {
                  "html": "./src/index.html",
                  "js": "./src/renderer.tsx",
                  "name": "main_window",
                  "preload": {
                    "js": "./src/preload.ts"
                  }
                }
              ]

I found it by looking at the repository related to another reported bug. It would be truly appreciated if you could update the documentation a little bit - I was searching for hours to find a way how tu use typescript in the preload script.

@MarshallOfSound is it possible to use another webpack config for the preload file?
My use case is I'm using the polllyfills in the renderer that I don't want in preload. Bc I'm using the node webpack attribute there's no option for excluding a file. Thank you!

@karlitos That was works for me to in "dev" electron-forge start script. But when I build distribution using @electron-forge/maker-squirrel electron-forge make do not loaded the preload.ts, it does not appear on .webpack folder.
My package.json:
/* other things*/

"plugins": [
        [
          "@electron-forge/plugin-webpack",
          {
            "mainConfig": "./webpack.main.config.js",
            "renderer": {
              "config": "./webpack.renderer.config.js",
              "entryPoints": [
                {
                  "html": "./src/index.html",
                  "js": "./src/renderer.ts",
                  "name": "main_window",
                  "preload": {
                    "js": "./src/preload.ts"
                  }
                }
              ]
            }
          }
        ]
      ]
    }

My index.ts
/* other things */
const createWindow = (): void => {
  const mainWindow = new BrowserWindow({
    height: 700,
    width: 720,
    minWidth: 560,
    minHeight: 560,
    webPreferences: {
      preload: MAIN_WINDOW_PRELOAD_WEBPACK_ENTRY, /* it Works on start, but in production does not */
    },
    backgroundColor: '#eee',
    title: `Fechamento | Abertura de Bonus | Dev Danilo Souza | Versão ${app.getVersion()}`,
    icon: path.join(__dirname, 'assets', 'logo.ico')
  });

  mainWindow.loadURL(MAIN_WINDOW_WEBPACK_ENTRY);
};
/* other things */

Can anybody help me?
@MarshallOfSound @karlitos
For now, thanks!

It only says that I should use the variable MAIN_WINDOW_PRELOAD_WEBPACK_ENTRY but says nothing where it comes from and what should I do to initialize it.
Also it'd be very helpful to explain the magic behind the naming.

@evil-shrike Yes, I am also in the same situation, no idea from where these variables are coming or where to define these! Any help will be appreciated.

Hey Guys, it really IS described in the docs, but very hidden.
https://www.electronforge.io/config/plugins/webpack#project-setup

In your package.json
under config->plugins->renderer->entrypoints-Array, you need to add to the objects the property "preload":{"js":./src/preload.ts"}  ( if your preload file is located at  src/preload.ts).
the entrypoints-array-objects have a name-propertygiven - in the default-template it has the name:"main_window"

if you DON'T define the preload-script in the package.json then you only have by default only one variable autogenerated - MAIN_WINDOW_WEBPACK_ENTRY.
if you DO define the preload-script in the package.json, there will be 2 variables autogenerated for you now ( as described in the docs):
MAIN_WINDOW_WEBPACK_ENTRY and MAIN_WINDOW_PRELOAD_WEBPACK_ENTRY - prefixed with MAIN_WINDOW because you defined the entrypoint-name as "main_window" in your package.json.
in order to use these variable now in your index.ts / main-process-script they have to be declared upfront:
<<(index.ts)>>
declare const MAIN_WINDOW_WEBPACK_ENTRY: string;
declare const MAIN_WINDOW_PRELOAD_WEBPACK_ENTRY: string;
...
const mainWindow= new BrowserWindow({
    webPreferences: {
      preload: MAIN_WINDOW_PRELOAD_WEBPACK_ENTRY,

}})

<<(package.json)>>
"config": {
    "forge": {
      "packagerConfig": {},
      "makers": [
        {
          "name": "@electron-forge/maker-squirrel",
          "config": {
            "name": "my_new_app"
          }
        },
        {
          "name": "@electron-forge/maker-zip",
          "platforms": [
            "darwin"
          ]
        },
        {
          "name": "@electron-forge/maker-deb",
          "config": {}
        },
        {
          "name": "@electron-forge/maker-rpm",
          "config": {}
        }
      ],
      "plugins": [
        [
          "@electron-forge/plugin-webpack",
          {
            "mainConfig": "./webpack.main.config.js",
            "renderer": {
              "config": "./webpack.renderer.config.js",
              "entryPoints": [
                {
                  "html": "./src/index.html",
                  "js": "./src/renderer.ts",
                  "name": "main_window",
                  "preload": {
                    "js": "./src/preload.ts"
                  }
                }
              ]
            }
          }
        ]
      ]
    }
  },

To be honest, I do not understand why this is not pre-generated. At least, it would be nice to have something like that in the generated index.ts:
// This allows TypeScript to pick up the magic constant that's auto-generated by Forge's Webpack
// plugin that tells the Electron app where to look for the Webpack-bundled preload code (depending on
// whether you're running in development or production).
//
// This constant is only defined if a bundle for preload has been generated. In order to get it generated,
// preload needs to be defined as entry point. E.g. add
// "preload": { "js": "./src/preload.ts" }
// to the "entryPoints" in your package.json.
// declare const MAIN_WINDOW_PRELOAD_WEBPACK_ENTRY: string;

authors: DaniloSouza19 G-Rath MR250655 MarshallOfSound david-jares evil-shrike fsdadnan jpilgrim karlitos

repo: electron-userland/electron-forge

license: mit


text:
afterPrune executed only with DEBUG=electron-packager

Pre-flight checklist

[X] I have read the contribution documentation for this project.
[X] I agree to follow the code of conduct that this project uses.
[X] I have searched the issue tracker for a bug that matches the one I want to file, without success.

Electron Forge version
7.4.0
Electron version
v31.3.1
Operating system
macOS 14.6.1
Last known working Electron Forge version
No response
Expected behavior
afterPrune should be executed when running npx electron-forge package --arch=arm64 --platform=linux ./electron
Actual behavior
the hook is only executed with DEBUG=electron-packager npx electron-forge package --arch=arm64 --platform=linux ./electron
Steps to reproduce
I have a react app in the main folder project and electron files into ./electron.
This is the forge.config.js file:
const { FusesPlugin } = require("@electron-forge/plugin-fuses")
const { FuseV1Options, FuseVersion } = require("@electron/fuses")

module.exports = {
    packagerConfig: {
        prune: true,
        afterPrune: [
            async (buildPath, electronVersion, platform, arch) => {
                throw new Error("This is a test error")
            },
        ],
    },
    rebuildConfig:{},
    makers: [
        {
            name: "@electron-forge/maker-zip",
            platforms: ["linux", "darwin"],
        },
    ],
    plugins: [
        new FusesPlugin({
            version: FuseVersion.V1,
            [FuseV1Options.RunAsNode]: false,
            [FuseV1Options.EnableCookieEncryption]: true,
            [FuseV1Options.EnableNodeOptionsEnvironmentVariable]: false,
            [FuseV1Options.EnableNodeCliInspectArguments]: false,
            [FuseV1Options.EnableEmbeddedAsarIntegrityValidation]: true,
            [FuseV1Options.OnlyLoadAppFromAsar]: true,
        }),
    ],
}

the error is not thrown unless you use DEBUG=electron-packager
Additional information
$ npx electron-forge package --arch=arm64 --platform=linux ./electron
✔ Checking your system
✔ Preparing to package application
✔ Running packaging hooks
  ✔ Running generateAssets hook
  ✔ Running prePackage hook
❯ Packaging application
  › Determining targets...
  ❯ Packaging for arm64 on linux
    ✔ Copying files
    ✔ Preparing native dependencies [5s]
    ⠸ Finalizing package
◼ Running postPackage hook

$ DEBUG=electron-packager npx electron-forge package --arch=arm64 --platform=linux ./electron
✔ Checking your system
❯ Preparing to package application
✔ Preparing to package application
❯ Running packaging hooks
❯ Running generateAssets hook
✔ Running generateAssets hook
❯ Running prePackage hook
✔ Running prePackage hook
✔ Running packaging hooks
❯ Packaging application
› Determining targets...
  electron-packager Electron Packager 18.3.4
  electron-packager Node v20.17.0
  electron-packager Host Operating system: darwin 23.6.0 (arm64) +0ms
  electron-packager Packager Options: {"asar":false,"overwrite":true,"ignore":[{}],"prune":true,"afterPrune":[null],"quiet":true,"dir":"<PATH>/frontend","arch":"arm64","platform":"linux","afterFinalizePackageTargets":[null],"afterComplete":[null],"afterCopy":[null],"afterExtract":[null],"out":"<PATH>/frontend/out","electronVersion":"31.3.1"} +1ms
  electron-packager Target Platforms: linux +0ms
  electron-packager Target Architectures: arm64 +0ms
  electron-packager Inferring application name from name in <PATH>/frontend/package.json +2ms
  electron-packager Inferring appVersion from version in <PATH>/frontend/package.json +0ms
  electron-packager Application name: myName +0ms
  electron-packager Target Electron version: 31.3.1 +0ms
  electron-packager Ignored path regular expressions: [
  /^\/out\//g,
  '/package-lock\\.json$',
  '/yarn\\.lock$',
  '/\\.git($|/)',
  '/node_modules/\\.bin($|/)',
  '\\.o(bj)?$',
  '/node_gyp_bins($|/)'
] +0ms
❯ Packaging for arm64 on linux
❯ Copying files
❯ Preparing native dependencies
❯ Finalizing package
  electron-packager Downloading Electron with options {"platform":"linux","arch":"arm64","version":"31.3.1","artifactName":"electron"} +521ms
  electron-packager Creating /var/folders/sh/yv5pz62x5c95fp81cq72cqlr0000gn/T/electron-packager/linux-arm64-template-hvxaJe +766ms
  electron-packager Extracting /Users/tommyblue/Library/Caches/electron/b87e5a986f1e10f6b1c405feb225bb1c8d2a6bcdf6509ea3217a07cb7f6bd118/electron-v31.3.1-linux-arm64.zip to /var/folders/sh/yv5pz62x5c95fp81cq72cqlr0000gn/T/electron-packager/linux-arm64-template-hvxaJe +1ms
  electron-packager Initializing app in /var/folders/sh/yv5pz62x5c95fp81cq72cqlr0000gn/T/electron-packager/tmp-GLizFU from /var/folders/sh/yv5pz62x5c95fp81cq72cqlr0000gn/T/electron-packager/linux-arm64-template-hvxaJe template +1s
  electron-packager Ignored paths based on the out param: [ '<PATH>/frontend/out' ] +1ms
✔ Copying files
✔ Preparing native dependencies

An unhandled rejection has occurred inside Forge:
Error: This is a test error
at module.exports.packagerConfig.afterPrune (<PATH>/frontend/forge.config.js:40:23)
    at node:internal/util:432:7
    at new Promise (<anonymous>)
    at node:internal/util:418:12
    at AsyncFunction.<anonymous> (<PATH>/frontend/node_modules/@electron-forge/core/dist/api/package.js:44:54)
    at node:internal/util:432:7
    at new Promise (<anonymous>)
    at AsyncFunction.<anonymous> (node:internal/util:418:12)
    at <PATH>/frontend/node_modules/@electron/packager/dist/hooks.js:9:73
    at Array.map (<anonymous>)
    at promisifyHooks (<PATH>/frontend/node_modules/@electron/packager/dist/hooks.js:9:29)
    at LinuxApp.copyTemplate (<PATH>/frontend/node_modules/@electron/packager/dist/platform.js:137:46)
    at async LinuxApp.buildApp (<PATH>/frontend/node_modules/@electron/packager/dist/platform.js:125:9)
    at async LinuxApp.initialize (<PATH>/frontend/node_modules/@electron/packager/dist/platform.js:120:13)
    at async LinuxApp.create (<PATH>/frontend/node_modules/@electron/packager/dist/linux.js:14:9)
    at async Promise.all (index 0)

I'm seeing this behaviour (hang on "Finalizing package") with an afterPrune hook that literally does nothing; just an empty async function. With DEBUG=electron-packager, my build completes successfully in ~5 seconds.
If I modify the function to call the callback parameter, which the current electron-packager docs do not indicate should be necessary, then the build completes.
Electron Forge version: 7.5.0
Electron version: 32.0.1
OS: Windows 11 Enterprise 23H2

Use the packageAfterPrune hook instead, it supports a simple async function --> https://www.electronforge.io/config/hooks#packageafterprune
The packager hooks all stil use callback syntax, you can avoid that by using the forge hooks that map to them

Ah, that is much nicer! Thank you.

authors: MarshallOfSound jeremypenner tommyblue

repo: electron/forge


text:
[ELE-58] Optionally combine slack alerts into a single message

Is your feature request related to a problem? Please describe.
When we have multiple test failures due to bad data making its way into our source systems or some coding mistake in a base model, edr monitor sends a slack message for each alert.
This creates a lot of noise in our slack channel, and can seem like a much bigger problem than it really is.
Describe the solution you'd like

Ability to configure grouping for slack notifications, default can remain the same but additional options to group by 'all' and 'table' would be great
Slack message body would be different, including a count of the alerts and listing the tables, etc. that are alerting.

Describe alternatives you've considered
Rolling my own slack notifier, but yours is already very good!
Would you be willing to contribute this feature?
Possibly, but my bandwidth will be limited in the short term.
ELE-58

Thanks for opening this @jakeberesford-palmetto!
Indeed sending an alert per failure could work when you have a small number of test / model failures. The big advantage for doing so is that each alert can be very detailed with tons of useful information when you do the troubleshooting. The downside as mentioned here is that it could be too much for when you all of sudden have a spike of failures due to incidents.
I think there could be several different approaches here -
Group alerts by errors - send a detailed alert per error with a list of relevant models / tests. This requires further scoping but the idea is to have a solution for spikes due to the same error.
Group alerts by table - if you have multiple tests running on the same table, send a list of failed tests for the same table in a single alert. This could help if you have lots of tests defined per table.
Group all alerts into a single alert - this can enable receiving a single alert for all the failures during the last run. The downside here is that the granularity for each error could be limited. To solve that we can attach the report and then further troubleshooting can be done using the report itself.
Would love to hear your thoughts!
I'll add it to the ticket as well. Would love to hear your thoughts about it

I think those groupings make a lot of sense. I can see value in all 3, do you think it'd be possible for elementary to determine the best grouping given a list of alerts?
Perhaps there might be a configurable threshold where tests are grouped? If I just have 1 or 2 alerts, the existing alerts are great, but if I have 3+ alerts I'd prefer them to be rolled up into a single alert.
For the aggregated alerts I'd want to see more of a summary, the number of alerts, a query to get specific alert details from the data warehouse, and a link to the edr report would be nice!

Hey @jakeberesford-palmetto - we added the ability to group alerts by table!
You can see the details on our docs under the "Group alerts by table" section.

authors: Hadarsagiv jakeberesford-palmetto oravi

repo: elementary-data/elementary


text:
Column size issue in 'all_columns_anomalies' test

Issue description:
On the test 'all_columns_anomalies' the results table is created using the first batch of results (the first column), and the results column sizes are assigned by the content of this batch.
If there is a following batch with larger values, the insert of the batch fails.
Solution:
Create the table with an empty template (like we do with the incremental tables).
The fix will be deployed as part of the Redshift integration branch:
https://github.com/elementary-data/dbt-data-reliability/pull/20

Happy to update that this issue was fixed with version 0.3.12 of the dbt package. Closing this issue 🥳

authors: Maayan-s oravi


text:
Fixed the bug that caused the force test not to work

Fixed the following bugs:

Long press doesn't work properly.
After the force test the variable calibrate_servo_no becomes 6, the calibration of servos with the A button starts with 6.

This pr has conflict with new version.If you have the same problem, please submit it again on a new basis

repo: elephantrobotics/myCobot


text:
fix rectangle point coordinate

Hi Edouard,
It seems there are two same points (w1v and w1v) in r1_points when checking collision. But they should be opposite at Y-axis (-w1v and w1v).

How could this bug stay unnoticed for so long? :scream:
Thank you very much!

authors: dongleecsu eleurent

repo: eleurent/highway-env


text:
solving asynchronization issue in AsyncTextToSpeechClient in both convert and convert_as_stream

Both convert and convert_as_stream functions had an issue while using the "AsyncElevenlabs" class. The issue appeared as follows:
when I tried to do the following:
client = AsyncElevenLabs(api_key=...) audio_stream = await client.generate(text="how are you", stream=True, ....) async for i in audio_stream:     print(I)
I got this error:
coroutine object doesn't support the asynchronous context manager protocol
The issue was solved by adding await within the "convert and convert_as stream" functions in the "AsyncTextToSpeechClient" class.

Ok @fern-bot thank you.

@fern-bot  I have a Code Formatter extension in my VS code that's why all these changes happened.
the added code in lines 383 and 519

I wanted to be added as contributor 😫

repo: elevenlabs/elevenlabs-python


text:
🛑 UNER - Guaraní is down

In b066113, UNER - Guaraní (https://g3autogestion.uner.edu.ar/g3w3/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: UNER - Guaraní is back up in 9c17d71 after 28 minutes.

repo: elfoche/monitoreo


text:
🛑 UNER - Campus is down

In bba039b, UNER - Campus (https://campus.uner.edu.ar) was down:

HTTP code: 0
Response time: 0 ms

Resolved: UNER - Campus is back up in cf352a3.


text:
Keep Downloading dialog disappearing before selecting

Go to Trakt -> MyProgress (Maybe from other place too)
Choose a torrent and play it.
Stop the playback (Make sure in the setting for asking 'Keep Downloading')

The stop process also have a little bug, it doesn't stop immediatly.

The "Keep Downloading?" dialog show up with the 'busy' circle on it, so I can't press on any button there.
After the 'busy' circle disappear, the dialog gone too, and the torrent also gone from the torrent list.

Kodi Log:
https://paste.ubuntu.com/p/JF7vbbS4k4
Image:

Also the buffer downloaded fast but the torrent is still big, and probably the download decrease so the playback start laggy.
Your Environment
Elementum 0.1.41
Kodi 17.6
Windows 10, 64 bit

@burekas7 in Elementum v0.1.41 there is a new option to autoclose confirm dialogs after timeout. You can disable it or increase the timeout in plugin settings.

The setting is actually called:
"Automatically answer Yes after timeout"
But here it chose the "No" option, and there wass no timeout, I just stop the playback as usual.

there wass no timeout

Timeout for answering the dialog.

the setting is actually called

Description is not corrent. If "yes" is predefined - treats as "yes". If "no" is predefined - treats as "no".

What do you mean by 'predefined', according to my skin setting?
In my skin settings there is a setting of:
"Set default focus to NO button in confirmation dialog" (which is false in my case, which means that the predefined should be "YES" and it still check it as "NO" and remove it from the torrent list)
But the big question is why there is a timeout when stopping the playback at the first place?
Why there is this 'busy' circle when stopping?
All the time I was stopping the playback and immedietly the 'Keep Downloading?' dialog was show up.
Currently when I stop, there is kind of a little 'stuck' (black screen), then it returns to the menu screen with the dialog and the 'busy' status, so I can't press on any button, and then it was disappearing (probably due to this new setting that you talk about)

@burekas7 is this still an issue?

No.
A long time pass and now with 0.1.62 it's ok (Even before I think)
Thanks.
Closing it.


text:
Add support for CoNLL-U data format

For compatibility with the Universal Dependencies treebanks, it would be great if bistparser could support the CoNLL-U data format[1], natively through a command line option. The format is almost identical to CoNLL-09 except for some column names are different, and:

Leave comments in the output
Skip segmentation spans like 2-4

2-4 dámelo
2    da
3    me
4    lo

And empty node spans like 5.1

1      Sue       Sue
2      likes     like
3      coffee    coffee
4      and       and
5      Bill      Bill
5.1    likes     like
6      tea       tea

http://universaldependencies.org/format.html

i made a test code for bist-parser to train UD treebanks.
https://github.com/dsindex/bist-parser
apply convert.py

Writing a conversion script would be one way of doing it, but I think a better way is to integrate the functionality natively, perhaps with a commandline switch --conllu. If you're doing a lot of work with dependency parsers (SyntaxNet, UDpipe, Bist, MST, Malt) it is really convenient if you don't have to have a suite of conversion scripts and you can just expect them to input/output the (now increasingly standard) CoNLL-U format. :)

Thank you for your detailed request and interest in BIST Parser. I have added support of conll-u format in the graph-based parser for now (the transition-based parser will be upgraded soon too). Could please check if it is working ok on your end? (Simply pass the conll-u files instead of the old conll file. The graph-based parser can now parse both conll and conll-u formats seamlessly).

Thanks! Your way of dealing with it is much more elegant than mine was going to be :)
I tried it, it seems like there are still some problems:
$ python src/parser.py --dynet-seed 123456789 --outdir /tmp --train ~/data/ud-treebanks-conll2017/UD_Finnish/fi-ud-train.conllu --dev ~/data/ud-treebanks-conll2017/UD_Finnish/fi-ud-dev.conllu --epochs 30 --lstmdims 125 --lstmlayers 2 --bibi-lstm
[dynet] random seed: 123456789
[dynet] allocating memory: 512MB
[dynet] memory allocation done.
Using external embedding: None
Preparing vocab
Traceback (most recent call last):
  File "src/parser.py", line 57, in <module>
    words, w2i, pos, rels = utils.vocab(options.conll_train)
  File "/home/UT/source/bist-parser/bmstparser/src/utils.py", line 34, in vocab
    for sentence in read_conll(conllFP):
  File "/home/UT/source/bist-parser/bmstparser/src/utils.py", line 54, in read_conll
    tokens.append(ConllEntry(int(tok[0]), tok[1], tok[2], tok[4], tok[3], tok[5], int(tok[6]) if tok[6] != '_' else -1, tok[7], tok[8], tok[9]))
ValueError: invalid literal for int() with base 10: 'Symb'

This diff seems to fix the problem:
$ git diff
diff --git a/bmstparser/src/utils.py b/bmstparser/src/utils.py
index 52bbf79..e9efff7 100644
--- a/bmstparser/src/utils.py
+++ b/bmstparser/src/utils.py
@@ -43,12 +43,12 @@ def read_conll(fh):
     root = ConllEntry(0, '*root*', '*root*', 'ROOT-POS', 'ROOT-CPOS', '_', -1, 'rroot', '_', '_')
     tokens = [root]
     for line in fh:
-        tok = line.strip().split()
-        if not tok:
+        tok = line.strip().split('\t')
+        if not tok or line.strip() == '':
             if len(tokens)>1: yield tokens
             tokens = [root]
         else:
-            if tok[0] == '#' or '-' in tok[0] or '.' in tok[0]:
+            if line[0] == '#' or '-' in tok[0] or '.' in tok[0]:
                 tokens.append(line.strip())
             else:
                 tokens.append(ConllEntry(int(tok[0]), tok[1], tok[2], tok[4], tok[3], tok[5], int(tok[6]) if tok[6] != '_' else -1, tok[7], tok[8], tok[9]))

I think the problem is because .split() splits on both tabs and spaces and CoNLL-U only allows tabs as column delimiters.

Thanks! I integrated the diff you suggested and I hope it is working now. I will apply the same changes to the transition-based parser, and hopefully you could use it with conll-u data very soon.

Added conll-u support for transition-based parsing too..
It seems to be working on my end. I would appreciate if you could check if it is working for you too.
Thanks again!

Great, thanks! :) I've just checked it and it seems to be working fine (at least it's training). I'll train a whole model and get back to you when I've tested it end to end.

The model trains fine, I get some errors with the eval script.
Use of uninitialized value $head_g in string eq at src/utils/eval.pl line 1456, <SYS> line 39536.
Use of uninitialized value $head_g in numeric le (<=) at src/utils/eval.pl line 1462, <SYS> line 39536.
Use of uninitialized value $head_s in string eq at src/utils/eval.pl line 1465, <SYS> line 39536.
Use of uninitialized value $head_s in string eq at src/utils/eval.pl line 1469, <SYS> line 39536.
Use of uninitialized value $head_s in numeric le (<=) at src/utils/eval.pl line 1475, <SYS> line 39536.
Argument "joka" isn't numeric in numeric le (<=) at src/utils/eval.pl line 1462, <SYS> line 39630.
Argument "joka" isn't numeric in numeric le (<=) at src/utils/eval.pl line 1475, <SYS> line 39630.
Use of uninitialized value $head_g in string ne at src/utils/eval.pl line 1446, <SYS> line 39630.
Use of uninitialized value $head_s in string ne at src/utils/eval.pl line 1446, <SYS> line 39630.
Use of uninitialized value $dep_g in string ne at src/utils/eval.pl line 1447, <SYS> line 39630.
Use of uninitialized value $dep_s in string ne at src/utils/eval.pl line 1447, <SYS> line 39630.
Use of uninitialized value $head_g in string eq at src/utils/eval.pl line 1452, <SYS> line 39630.

The prediction also seems to work fine:
$ wc -l ~/data/ud-treebanks-conll2017/UD_Finnish/fi-ud-dev.conllu
22403 /home/UT/data/ud-treebanks-conll2017/UD_Finnish/fi-ud-dev.conllu
$ wc -l /tmp/test_pred.conll
22403 /tmp/test_pred.conll

I changed the eval script for conll-u from the old eval script to the new script used by the conll 2017 shared task. The error messages you experienced should disappear now.

All works on my end :) I had to hack the transitions to disallow multiple roots (see multiple root issue #10) so that the ud eval does not fail but all else works fine! Elegant solution as Fran said :)

authors: dsindex elikip ftyers mdelhoneux

repo: elikip/bist-parser


text:
Fix a typo in the docs

Just a minor typo fix I spotted when looking in the docs.
Thanks for this project, enjoying using it so far!

Thank you for catching this!

repo: elimintz/justpy


text:
Updated version of dependency-check-maven to 6.2.2

Updated pom.xml

Running "mvn site" with version 5.3.2 generates the following error :
"[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on project parent: Error generating dependency-check-maven:5.3.2:aggregate report: One or more exceptions occurred during dependency-check analysis: One or more exceptions occurred during analysis:
[ERROR]         Unable to resolve system scoped dependency: com.sun:tools:jar:1.8.0:system"
Updating to 6.2.2 resolves that.


text:
Multi-processing

I am currently working on making this project multithreaded, as I have many folders with tens of thousands of images(perhaps 100k+), and am wanting a slightly faster option.
Opening this as a means of communication.  If you have a discord account/email that would work better, as I will likely see that before a github issue comment.
My discord account is thecodingchicken#4835 if you would prefer to reach out there.

Hi @thecodingchicken,
Thank you very much for reaching out and for sharing your feedback!
We already received a few requests similar to yours, and therefore decided to implement batch processing in the next upcoming release of difPy. Therefore, with this there will also be an option to multi-thread the process. This release is in ongoing development, and there is no confirmed release date yet. Please therefore, be patient and soon processing on difPy will become a lot more flexible, and scalable. :-)
Again thanks and all the best,
Elise

Just an update.  Code is almost working, just have to sort out a few minor issues that break everything.  Seems to be the norm for me, as I haven't really done any coding in years.
Glad to hear that you are working on it as well.  Is it multithreaded in another language?  As I know the GIL prevents you from running many threads that are CPU-bound.  Hence why multiprocessing seems to be the only option for me, assuming that you stick in CPython, as compared to IronPython or Jython

Hi @thecodingchicken,
Great to hear you are working on it as well and that you're almost done! Congrats. Feel free to open a pull request when you're done - I would love to see what approach you took.
I am working on it as well yes, but as I have currently quite a few other things I am working on, it will still take a bit to be finalized. I am planning to use the Python multiprocessing library, as well as implement the ability to input images to difPy in batches, and save the process state so that it can be resumed if interrupted.
All the best and happy coding!
Elise

Hi @thecodingchicken,
I'm happy to let you know that difPy v4 now comes with multiprocessing. Thanks for your suggestion!
All the best,
Elise

authors: elisemercury thecodingchicken

repo: elisemercury/Duplicate-Image-Finder


text:
parent_as/1 does not work in the RHS of union_all/2

Elixir version
Elixir 1.14.4 (compiled with Erlang/OTP 24)
Database and Version
PostgreSQL 11.13
Ecto Versions
Ecto 3.10.3
Database Adapter and Versions (postgrex, myxql, etc)
Ecto SQL 3.10.1 / Postgrex 0.17.3
Current behavior
When using union_all/2 parent_as/1 is only available in the LHS query of the union.
  # This query is a bit nonsensical it was contrived to show the union_all bug but its a pointless query.
  def ecto_bug() do
    query_a =
      from(tm in TeamMember)
      |> select([tm], tm.member_id)
      |> where([tm], tm.team_id == parent_as(:t).id)

    query_b =
      from(tm in TeamMember)
      |> select([tm], tm.member_id)
      |> where([tm], tm.team_id == parent_as(:t).id)

    query_c = union_all(query_a, ^query_b)

    query = from(t in Team, as: :t, where: t.id in subquery(query_c))
    Repo.all(query)
  end

This the error I get from the function above:
** (Ecto.SubQueryError) the following exception happened when compiling a subquery.

    ** (Ecto.QueryError) could not find named binding `parent_as(:t)` in query:

    from t0 in TeamMember,
      where: t0.team_id == parent_as(:t).id,
      select: t0.member_id

The subquery originated from the following query:

from t0 in Team,
  as: :t,
  where: t0.id in subquery(
  #Ecto.Query<from t0 in TeamMember, where: t0.team_id == parent_as(:t).id, union_all: (from t0 in TeamMember,
  where: t0.team_id == parent_as(:t).id,
  select: t0.member_id), select: %{member_id: t0.member_id}>
),
  select: t0

    (elixir 1.14.4) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir 1.14.4) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto 3.10.3) lib/ecto/repo/queryable.ex:211: Ecto.Repo.Queryable.execute/4
    (ecto 3.10.3) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3

If I comment out where([tm], tm.team_id == parent_as(:t).id) from query_b it works fine.
Expected behavior
I would expect to be able to access the parent bindings from both sides of the union_all.

transferring to ecto

Thanks a lot @greg-rychlewski!!!!

Thanks for the report :)

authors: greg-rychlewski marcelloma

repo: elixir-ecto/ecto


text:
Support Ecto.Changeset.foreign_key_constraint/3 ?

I have a relational setup that I would like to just give me an {:error, changeset} tuple back for when the foreign record doesn't exist. However, no matter what I put I always get this error:
14:02:56.563 [error] Task #PID<0.559.0> started from #PID<0.557.0> terminating
** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    *  (foreign_key_constraint)

If you would like to stop this constraint violation from raising an
exception and instead add it as an error to your changeset, please
call `foreign_key_constraint/3` on your changeset with the constraint
`:name` as an option.

The changeset defined the following constraints:

    * intervals_schedule_id_fkey (foreign_key_constraint)

...

If I'm understanding right, SQLite knows there is a constraint here, but its unnamed and so all we get back to help is an empty *  (foreign_key_constraint) clue. And since there is no name to match on, Ecto can't reconcile it and things just crash. I've even tried using the name: "" option, but I think Ecto just ignores the empty string.
Is it even possible to get around this or are we limited by the abilities of SQLite here? I do understand I can use the foreign_key: :off option in the adapter to just skip validations entirely which is currently my workaround, but I would love to get the {:error, changeset} return instead.
Caveat: I am in no way a DB expert and this could be 100% user error. My migrations are simple and would love to know if Im just doing things wrong:
create table(:schedules) do
  add :name, :string
end

create table(:intervals) do
  add :schedule, references(:schedules)
end

Yeah this is a SQLite3 issue. See this issue for more details: https://github.com/elixir-sqlite/ecto_sqlite3/issues/28
We have a hack that works for some cases. I believe I tried adapting that hack for foreign keys in the above issue, but failed.
I encourage you to post a request on the SQLite3 forum asking for some sort of API from SQLite3 so we can support this! :smile:
In the meantime, I'll try thinking about whether there may be other hacks we can do. I'll play around a bit an re-open this if I find a new approach. I'll also update the docs to make this a known limitation.

For reference, here is what we get back from SQLite3 and thus why we cannot really do anything:
sqlite> pragma foreign_keys=on;
sqlite> create table a(b integer not null primary key);
sqlite> create table c(d references a(b));
sqlite> insert into c values(1);
Error: FOREIGN KEY constraint failed

No mention of which constraint failed. See https://sqlite.org/forum/forumpost/b17fdd0d59 for examples on other types of constraints.

I've added some documentation about this limitation that points back to this issue.

Hi @jjcarstens, sorry for this 2 years+ later ping but I'm considering Turso database (SQLite-based database) and I ended up on this thread while reading the limitations of SQLite3 support with Ecto.
Should I understand that even now foreign keys are not properly supported? Did you end up with a viable alternative or disabling foreign keys was the only solution?

My use-case had flexibility so I ended up disabling foreign keys when able and capturing and transforming the error when absolutely needed. Definitely a "made it work and moved on" situation. Hope that helps

Sorry for the post on a old thread, but just got this error on a project I'm working, and made a post on SQLite forumns using the same examples suggested by @kevinlang .
Hope they change it somehow.
https://sqlite.org/forum/forumpost/7a67578b26

Yes I hope they change it as well, but I doubt they'll add that since it has been spitting out this error for a very long time and some software expects that exact messaging. We'll see.

authors: binajmen jjcarstens kevinlang vitordahora-capital warmwaffles


text:
fix!: improve slug generation

will not remove periods from the permalink
will turn underscores into dashes
day and month will be padded with 0

Current dependencies on/for this PR:

main

PR #51   👈

This stack of pull requests is managed by Graphite.

repo: elixir-tools/tableau


text:
Joken.Config get_current_time and validate_time_claim calls updated …

… to Joken.Helpers module for update on Joken
https://github.com/bryanjos/joken/commit/c72001a0b15717e3bc60a4c0ebc2acae86f51b20

Thanks for that.
I've also updated the dependencies, and I've just updated the Openmaize hex package.

authors: riverrun robinsound

repo: elixircnx/openmaize


text:
Implement Tail Call Recursion

Connects to #221
Elixirscript does not support tail call recursion. The connected issue shows the steps that have been taken to try to get it working.
This issue is to either

[ ]  Implement Tail Call Recursion in Elixirscript directly

OR

[ ] Implement Tail Call Recursion here, the library used to make pattern matching functions

There are some articles, posts, and projects found to implement Tail Call Recursion and Optimization in JavaScript using trampolining. I've tried a couple of techniques, but haven't found anything that works that doesn't require rewriting how functions are written. I also am not sure if the way that pattern matched functions are created have any negative effects.
This can be considered done when the example in #221 doesn't throw a stack error

I believe that implementing it in Tailored would be better; the example code in the README already shows a recursive definition of the Factorial function which you'd like to be tail-recursive.

but haven't found anything that works that doesn't require rewriting how functions are written.
I am fairly certain that this indeed is the case. I expect that there is a mistake in https://github.com/elixirscript/tailored/blob/master/src/tailored/defmatch.js#L38 and that when we fix it, it will work. I'll investigate in more detail once I have the time the next few days.

I dug a little bit, and, as far as I can understand, to trampolines to work (or any stack allocation elimination), functions should be called in tail call position. And to implement refactoring to tail call position is a kinda hard problem with grunt solution, especially for mutually recursive stuff.
I googled a bit, and found https://github.com/elixir-lang/elixir/issues/6251 which means Elixir itself does have that kind of stuff. I tried to search for tail call elimination phase in Elixir source code and didn't succeed. But I believe going for that transformation is a way to go with TCO.
That is my observation, and I don't have Elixir or BeamVM experience at all to know if it's possible to translate that kind of stuff for Javascript.

Maybe the new format of how functions are compiled will help here. Instead of calling a function to produce a function the does pattern matching, now an actual named function is created with if statements for each function pattern. There is a part of me that believes because the old way dealt with a bunch of anonymous functions so it was possible that may have effected the stack.

Was able to implement this thanks in part to the new function translation

authors: Qqwy bryanjos troglotit

repo: elixirscript/elixirscript


text:
Fix random things

Still trying to prep a 1.1 B1  ... needs moar work

heh...
Apparently the testing of the headers is slightly too picky...
It should be enough to test for \s (spaces) instead of the exact sequence of spaces and tabs:
https://github.com/elkarte/Elkarte/blob/development/tests/sources/FilesTest.Basic.php#L110
https://github.com/elkarte/Elkarte/blob/development/tests/sources/FilesTest.Basic.php#L111
and so on... would you like to do the honors? :innocent:

LOL ... I was doing just that when you entered the comment ... I used /s+? which should cover spaces and tabs, any number, any combo ... lets see what it thinks now

Worked .. Travis gave me a :trophy:

In that one as well! :tada:

repo: elkarte/Elkarte


text:
Update README.md with antigen instruction.

Add Antigen plugin manager instruction.

thanks for this! just a couple of suggested formatting changes :)

authors: DanielAtCosmicDNA ellie

repo: ellie/atuin