text:
Visual Studio 2022 Build Issue

Hi, I am new to both Visual Studio and Openfast, and I ran into an issue regarding compiling openfast in Visual Studio.
I followed the documentation and installed all the dependencies, but when I tried to build the openfast in Visual Studio, here is what happened:

The corresponding output is like below:

It seems like I have an issue opening the input file "opencv_world401d.lib". Does anyone know how to solve it? Appreciate it.

Which version or git hash of OpenFAST?

I think that's the latest version of OpenFAST v3.5.2, I just cloned all the code from "https://github.com/OpenFAST/openfast".

This is a really strange error that I haven't seen before.  If you compile for 64 bit, do you get the same errors?
The link error about opencv_world401d.lib is puzzling.  This isn't a library that is used directly in OpenFAST.  I suspect it a backend library the compiler is using, and is either not available for 32 bit compiles, or there is something wrong with the compiler install.
Tagging @deslaughter  in case he has seen this before.

Which C/C++ compiler are you using? You could try to use a different one: for example, right-click on the 'FAST_Registryproject, clickProperties, then Configuration Properties->General->Platform Toolset`.

Note: I have old versions of Visual Studio and Intel installed, so you will have different options.

@andrew-platt I agree that it's a very strange issue. I haven't seen it before. It seems like something in Visual Studio has changed and is being applied to the OpenFAST projects.
@DavidChen0429 is this a fresh install of Visual Studio? Is it on a shared computer when the configuration may have been modified?

@DavidChen0429, which version of Windows are you running?
@deslaughter, at some point when I have free time, I'll upgrade VS on my Win10 VM and see what chaos that presents.

@andrew-platt I agree that it's a very strange issue. I haven't seen it before. It seems like something in Visual Studio has changed and is being applied to the OpenFAST projects.
@DavidChen0429 is this a fresh install of Visual Studio? Is it on a shared computer when the configuration may have been modified?

Yep. I'm using the latest version of Visual Studio that I downloaded a few days ago.

@DavidChen0429, which version of Windows are you running?
@deslaughter, at some point when I have free time, I'll upgrade VS on my Win10 VM and see what chaos that presents.

I'm using Windows 11 (x64). Is it possible that OpenFAST doesn't support Windows 11?

This is a really strange error that I haven't seen before. If you compile for 64 bit, do you get the same errors?
The link error about opencv_world401d.lib is puzzling. This isn't a library that is used directly in OpenFAST. I suspect it a backend library the compiler is using, and is either not available for 32 bit compiles, or there is something wrong with the compiler install.
Tagging @deslaughter in case he has seen this before.

That's the strange part, I selected "release" and "x64", but still the compiler worked as "Realse Win32" which you can see from the output. Is there a way to change that?

That's the strange part, I selected "release" and "x64", but still the compiler worked as "Realse Win32" which you can see from the output. Is there a way to change that?

You can change that from the Build->Configuration Manager window:

Change the Win32 to x64 for the FAST_Registry project. It was originally set so that we didn't have to keep rebuilding the registry for different configurations when we would normally build for Win32, but it shouldn't matter.

Hey guys, thanks for the tips, and the problem is solved by compiling for 64 bit as @andrew-platt suggested.
To be specific, I changed the platform of the FAST_Registry project from Win32 to x64 as the previous post from @bjonkman
showed. After doing so, I successfully built the solution by using the default compiler from the VS2022.

Thanks for the update!
For our next release, I think we should change the FAST_Registry to build only x64 as the necessary Win32 libraries are probably no longer installed by default.

authors: DavidChen0429 andrew-platt bjonkman deslaughter

repo: OpenFAST/openfast


text:
needs update for string.data changes

After JuliaLang/julia#19449, soon to be merged for Julia 0.6, you will no longer be able to access the raw bytes of a string via string.data; instead, do Vector{UInt8}(string), and instead of pointer(string.data) just do pointer(string).  For example, this affects:
https://github.com/OpenGene/HTSLIB.jl/blob/343cf654e2e4b89f0b8c7f60e61dc7672a28bd6d/src/bam/bam.jl#L32
At first glance, it looks like the KString type is rather unsafe: it takes a string argument and stores a pointer to its underlying data.  But if the original string is garbage-collected, then this pointer will become invalid.  Would be better to store a reference to the actual string object.  Or just use an existing type like SubString, which seems very similar to what you are trying to do with KString?

@zhmz90 could you please help to update?

source: gharchive/issue

authors: sfchen stevengj

license: mit

license_type: permissive

license_source: bigquery


text:
Roundabouts are rendered with grey filling

Bug description
What is happening?
Enclosed ways 198571047 and 198571048 are rendered with grey filling
What should be happening?
Maybe render without filling, currently it implies being an area. Jeff asked to file report.
Repro Steps
Steps to reproduce the behavior:

Go to #map=17/59.42303/24.78610
Set year between 1970-1990
See error

Screenshots & URL
URL link is listed in Repro section's step 1.

Note: some ways weren't rendered yet when screenshot was taken.

Thanks for this! Because these are closed rings, our cartographic styling treats them as "transport areas." I am going to filter these areas out of the styling in the next round of updates.

Forthcoming styling:

are those for dirt roads, etc.?

This would be for any closed transit line that does not have an area=yes tag. And that is a tag that we will want exposed in the tiles to be able to symbolize actual transit areas.

From https://github.com/OpenHistoricalMap/issues/issues/205#issuecomment-826027674, area=yes was added in the Vtiles -  Staging.

This is fixed, clossing here!

authors: Rub21 jeffreyameyer kallejre tsinn

repo: OpenHistoricalMap/issues

license_source: github-api


text:
Need to fix the copy on https://staging.openhistoricalmap.org/copyright

Bug description
This page: https://staging.openhistoricalmap.org/copyright
has a lot of things that need to be cleaned up.
** Acceptance criteria **

[x] No dittos or placeholder copy
[ ] All links work - i.e. connect to live, up to date pages

Checked all links and they work.
Note that https://wiki.openstreetmap.org/wiki/Open_Historical_Map/License is a stub

authors: danrademacher jeffreyameyer


text:
decimaldate.js doesn't pass osm-website pre-push & pre-commit checks

Bug description
After an upstream merge of osm-website code it's no longer possible to commit the incorporated decimaldate.js code due to pre-push and pre-commit linting.
What should be happening?
Should be able to commit this code to our repository.
Screenshots & URL
Running pre-commit hooks
Check for trailing whitespace....................[TrailingWhitespace] OK
Analyze with ESLint..........................................[EsLint] FAILED
Errors on modified lines:
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 9, col 34, Error - Unnecessary escape character: \-. (no-useless-escape)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 9, col 46, Error - Unnecessary escape character: \-. (no-useless-escape)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 9, col 54, Error - Unnecessary escape character: \-. (no-useless-escape)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 18, col 20, Error - Missing radix parameter. (radix)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 19, col 18, Error - Missing radix parameter. (radix)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 20, col 27, Error - Expected '===' and instead saw '=='. (eqeqeq)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 20, col 30, Error - Strings must use doublequote. (quotes)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 20, col 41, Error - Missing radix parameter. (radix)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 20, col 64, Error - Missing radix parameter. (radix)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 21, col 34, Error - Multiple spaces found before '// ISO 8601 sh...'. (no-multi-spaces)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 50, col 5, Error - 'yearint' is not defined. (no-undef)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 52, col 5, Error - 'yearint' is not defined. (no-undef)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 52, col 50, Error - Multiple spaces found before '// ISO 8601 sh...'. (no-multi-spaces)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 57, col 38, Error - 'yearint' is not defined. (no-undef)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 68, col 41, Error - 'yearint' is not defined. (no-undef)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 80, col 55, Error - Strings must use doublequote. (quotes)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 81, col 51, Error - Strings must use doublequote. (quotes)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 83, col 7, Error - 'yearint' is not defined. (no-undef)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 83, col 33, Error - 'yearint' is not defined. (no-undef)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 83, col 64, Error - Strings must use doublequote. (quotes)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 83, col 69, Error - Multiple spaces found before '// just the ye...'. (no-multi-spaces)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 84, col 12, Error - 'yearint' is not defined. (no-undef)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 84, col 20, Error - Expected '===' and instead saw '=='. (eqeqeq)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 84, col 50, Error - 'yearint' is not defined. (no-undef)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 84, col 86, Error - Strings must use doublequote. (quotes)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 84, col 92, Error - Multiple spaces found before '// BCE offset ...'. (no-multi-spaces)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 85, col 21, Error - Strings must use doublequote. (quotes)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 85, col 37, Error - 'yearint' is not defined. (no-undef)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 85, col 73, Error - Strings must use doublequote. (quotes)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 85, col 79, Error - Multiple spaces found before '// BCE offset ...'. (no-multi-spaces)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 92, col 15, Error - Expected '!==' and instead saw '!='. (eqeqeq)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 92, col 18, Error - Missing radix parameter. (radix)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 92, col 47, Error - Expected '===' and instead saw '=='. (eqeqeq)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 93, col 16, Error - Expected '!==' and instead saw '!='. (eqeqeq)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 93, col 19, Error - Missing radix parameter. (radix)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 94, col 14, Error - Expected '!==' and instead saw '!='. (eqeqeq)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 94, col 17, Error - Missing radix parameter. (radix)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 129, col 11, Error - Multiple spaces found before '// February'. (no-multi-spaces)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 139, col 11, Error - Unexpected trailing comma. (comma-dangle)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 154, col 15, Error - Expected '!==' and instead saw '!='. (eqeqeq)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 154, col 18, Error - Missing radix parameter. (radix)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 154, col 47, Error - Expected '===' and instead saw '=='. (eqeqeq)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 160, col 33, Error - Expected '===' and instead saw '=='. (eqeqeq)
/Users/erictheise/Repos/OpenHistoricalMap/ohm-website/app/assets/javascripts/decimaldate.js: line 160, col 59, Error - Expected '!==' and instead saw '!='. (eqeqeq)

Does the linter have a way to exempt certain files? If not, maybe we could pull in maplibre-gl-dates as an external dependency…

I disabled rules in eslint.js and erb-lint.yml so that the commit would go through but I think for the .js files the concerns are valid and not hard to fix. I'm not sure the issues flagged in ohm.style.* are all valid.


text:
LTE-FPV version with usb-modeswitch

remove wfb-ng and more packages
add usb_modeswitch
changed /etc/network/interfaces
changed /packages/majestic/majestic.mk (fix for compile as fpv)
tested at huawei e3372h

When compiling a new config, I ran into a situation where one of the name elements must be among ultimate , lite or fpv and it must be located last. This was about majestic.mk, although I still had to make corrections. But a number of commands for trimming the defconfig name, as I understand it, expect a mandatory part like fpv to be located last.
The very name of the config indicates that the assembly is intended for use in conjunction with a 4G LTE modem and at the same time with FPV, more precisely as a blank for such use. In fact, the majestic streamer and a number of tunnels / vpn remained there. And the means for the operation of the e3372 modem were integrated.

source: gharchive/pull-request

repo: OpenIPC/firmware

license: MIT


text:
Linking Issues on Windows with Visual Studio 2017 BuildTools

Problem
Hi,
I have linking issues on Windows using Visual Studio 2017 BuildTools.
Expected behavior:
Linking should occur without errors.
Actual behavior:
Linking occurs with errors:
"D:\Documents\Development\ThirdParty\openimageio\build\oiio\INSTALL.vcxproj" (default target) (1) ->
"D:\Documents\Development\ThirdParty\openimageio\build\oiio\ALL_BUILD.vcxproj" (default target) (3) ->
"D:\Documents\Development\ThirdParty\openimageio\build\oiio\src\libOpenImageIO\OpenImageIO.vcxproj" (default target) (5) ->
(Link target) ->
  jpeg.lib(jerror.c.obj) : error LNK2019: unresolved external symbol __imp___iob_func referenced in function output_message [D:\Documents\Development
\ThirdParty\openimageio\build\oiio\src\libOpenImageIO\OpenImageIO.vcxproj]
  tiff.lib(tif_unix.obj) : error LNK2001: unresolved external symbol __imp___iob_func [D:\Documents\Development\ThirdParty\openimageio\build\oiio\src
\libOpenImageIO\OpenImageIO.vcxproj]
  tiff.lib(tif_unix.obj) : error LNK2019: unresolved external symbol __imp_vfprintf referenced in function unixWarningHandler [D:\Documents\Developme
nt\ThirdParty\openimageio\build\oiio\src\libOpenImageIO\OpenImageIO.vcxproj]
  tiff.lib(snprintf.obj) : error LNK2019: unresolved external symbol __imp__vscprintf referenced in function _TIFF_vsnprintf_f [D:\Documents\Developm
ent\ThirdParty\openimageio\build\oiio\src\libOpenImageIO\OpenImageIO.vcxproj]
  tiff.lib(snprintf.obj) : error LNK2019: unresolved external symbol __imp__vsnprintf_s referenced in function _TIFF_vsnprintf_f [D:\Documents\Develo
pment\ThirdParty\openimageio\build\oiio\src\libOpenImageIO\OpenImageIO.vcxproj]
  D:\Documents\Development\ThirdParty\openimageio\build\oiio\src\libOpenImageIO\Release\OpenImageIO.dll : fatal error LNK1120: 4 unresolved externals
 [D:\Documents\Development\ThirdParty\openimageio\build\oiio\src\libOpenImageIO\OpenImageIO.vcxproj]

Steps to Reproduce
I'm running with a very minimalist set of dependencies right now and the command I issue is as follows:

cmake D:\Documents\Development\ThirdParty\openimageio\oiio -DCMAKE_INSTALL_PREFIX=D:\Documents\Development\ThirdParty\openimageio/build\release -DILMBASE_INCLUDE_PATH=C:\Users\thomas\Packages\ilmbase\2.2.0\ilmbase\include -DOPENEXR_INCLUDE_PATH=C:\Users\thomas\Packages\openexr\2.2.0\openexr\include -DBOOST_ROOT=C:\Users\thomas\Packages\boost\1.64.0\boost -G "Visual Studio 15 Win64" -T v141,host=x64

Here is a Gist with the full output: https://gist.github.com/KelSolaar/d27ee328824fc7a079fdfbbc28b9d03d
Versions

OIIO branch/version: 1.7.15
OS: Windows 10
C++ compiler: Visual Studio 2017 BuildTools - cl

Just to confirm that same happens with https://github.com/OpenImageIO/oiio/releases/tag/Release-1.8.4dev

repo: OpenImageIO/oiio


text:
simd exp, log, pow (exact and approximate/fast versions)

simd exp, log, pow (exact and approximate/fast versions)
Also add code to simd_test to not only verify that they work, but also
benchmark them.

Looks good otherwise, are you able to hit a 4x speedup in microbenchmarks?

It's a complex landscape, easier to just report results than try to characterize it simply.
Here's what I'm getting on my laptop. This is without fma enabled, which really speeds up the madds.
The ones without "simd" in the name are scalar.
expf:   171.2 Mvals/sec
fast_exp:   135.3 Mvals/sec
simd::exp:   397.6 Mvals/sec
simd::fast_exp:   635.7 Mvals/sec
logf:   169.0 Mvals/sec
fast_log:   167.6 Mvals/sec
simd::log:   358.3 Mvals/sec
simd::fast_log:   627.4 Mvals/sec
powf:    64.1 Mvals/sec
simd::fast_pow:   252.3 Mvals/sec

Wow - that seems to be super-linear in some cases! (Though I'm sure the way is benchmarked might have something to do with that).
fast_safe_pow is doing more than just looking at sign(x) -- its also checking if y is an odd or even integer. Otherwise pow(-3.0, 2.0) would be wrong. For fractional powers it returns 0 for negative x (so that pow(-3.0, 0.5) works as expected).
But this is all getting closer to more of a safe_pow type of behavior. The question is what the use case will be. If x is always going to be positive, you might as well just use exp(log(x)*y) directly and not even bother making a wrapper?

It looks like some platforms are having trouble the inline asm incantation in the benchmark script.

inline asm -- Yes, I have a fix for that in progress. :-)

Crap, do I still have pow wrong?
The use case I had in mind for short-term use was actually linear<->sRGB conversion. So I'm really only concerned with positive x inputs (or at least, clamping x to a minimum of 0 is fine).
I could get rid of the full-precision simd::pow for now, return to it if we need it.
I could rename fast_pow to fast_pow_positive or something, and document it as working only for positive inputs?
Or leave the exp and log, and leave it to the individual use cases how careful they want to be. Ugh.

Or should I try to fully duplicate fast_safe_pow in simd?
I can try that and see how bad the slowdown is with all the special case handling. But I wanted something really fast.

Thats what I suspected the next use-case would be. My vote would be to just implement exp and log and use those directly for the sRGB math.

I think the corner cases in pow really preclude a super fast implementation. So you would either end up with lots of variants or making assumptions that don't fit with some use cases.
So I think leaving it out is acceptable.
For OSL (fast_safe_pow), we already have those somewhat "loose" math semantics, so it was easier to bake in some assumptions (although there is still definitely a cost to them). But its definitely just wasted cycles for something like gamma conversion.

Yes, I think you are correct. So my plan is:

remove simd pow and safe_pow, since they are not correct.

use the clamp/exp/log directly in the srgb code.

if it comes up in other contexts that make us think we should have something in simd.h or fmath.h, we can cross that bridge when we come to it.

Sounds good!

Amended as follows:

Removed full-precision pow, which was incorrect.

Altered and renamed approximate pow that only works for positive inputs: fast_pow_pos(). This can be handy a number of places.

Amended this review with the changes that were the original use case, which is SIMD evaluation of sRGB <-> linear conversion.

LGTM

authors: fpsunflower lgritz


text:
[BUG] this issue arises only when there are many nested views using 'with .. as' in the program.

When I was using version 1.7.0 of opnelineage, I encountered this issue. Not all Spark programs encounter this problem now. I found that this issue arises only when there are many nested views using 'with .. as' in the program.
My Spark application works in a Yarn Hadoop cluster. After completing its tasks and attempting to close the SparkContext, my application encounters an error:
ERROR util.Utils: Uncaught exception in thread shutdown-hook-0
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1257)
at java.lang.Thread.join(Thread.java:1331)
at org.apache.spark.scheduler.AsyncEventQueue.stop(AsyncEventQueue.scala:135)
at org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
at org.apache.spark.scheduler.LiveListenerBus$$anonfun$stop$1.apply(LiveListenerBus.scala:219)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.scheduler.LiveListenerBus.stop(LiveListenerBus.scala:219)
at org.apache.spark.SparkContext$$anonfun$stop$6.apply$mcV$sp(SparkContext.scala:1922)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1361)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1921)
at org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:573)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1992)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

I speculate that the possible reason is that the parsed Spark event queue is too long, leading to timeout during consumption, resulting in an error when closing the event queue.

Thanks @zhangyuewen12, I don't think we solved this yet in newer versions. Your description on how to reproduce this seems to be highly relevant. Thank you for providing this.
In recent versions of OpenLineage, we included circuit breaker mechanism (https://openlineage.io/docs/integrations/spark/configuration/circuit_breaker) with timeout (that was added recently). It is optional and you can set it to perhaps ~3mins. In this case OL integration code will stop after defined amount of this which you may find helpful in mitigating the problem you described.
I hope we'll find the solution for the root cause in the future too.

repo: OpenLineage/OpenLineage


text:
Python: enable deserialization to Python classes

Problem
There's no out-of-the-box solution that would allow deserialize OL events into Python classes. As the continuation of the effort made to generate Python facets out of JSON schemas this PR aims to enable such serialization.
Solution

Use old type annotations to enable Python 3.8 compatibility.
Add global variable that registers and holds keys under which facets should be stored in the structure.
Add logic to properly deserialize nested types into proper classes.

Few additional points

attrs are limited to accept attributes names that Python can assign into variables. Therefore, e.g. for my-facet.great key in event payload this is going to be sanitized into my_facet_great and original named would be stored in attr.field(metadata={"name":"my-facet.great"})
above lacks tests
for unresolved types there are basic types created with additional attributes (e.g. run->facets->custom would be RunFacet with some additional attributes)
one can register their own custom facets using utils.register_facet_key to enable richer deserialization
after couple approaches Pydantic didn't turn out to solve issues with deserializing nested fields into proper classes. e.g. anything under run->facets goes into generic RunFacet. After some discussion with @mgorsk1 I still don't see clear way how Pydantic could be significantly better for deserialization task. https://github.com/OpenLineage/OpenLineage/pull/2630#discussion_r1605762463 might work but is complicated and does not leverage to use Pydantic instead

Checklist

[x] You've signed-off your work
[ ] Your pull request title follows our guidelines
[ ] Your changes are accompanied by tests (if relevant)
[ ] Your change contains a small diff and is self-contained
[ ] You've updated any relevant documentation (if relevant)
[ ] Your comment includes a one-liner for the changelog about the specific purpose of the change (not required for changes to tests, docs, or CI config)
[ ] You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
[ ] You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2024 contributors to the OpenLineage project

True, pydantic approach, cleaner as it is, hit the same issue attrs approach had and after working on it for quite a while I also did not found quick solution. However, what I think is the root origin of all these issues (both attrs/pydantic) is OpenLineage.json model. For example, the way events are modeled now, there is no connection between any of the facet (like ParentRunFacet, ErrorMessageFacet) and Run.facets object. That's why any approach to turn this model into attrs/pydantic/dataclasses models will require custom workarounds.

What's next? Hope this PR can be reviewed soon.

@JDarDagran does this PR works with Airflow without any particular changes?

In what way it could work/not work with Airflow?
It sure needs extra deserialize - I don't think we want to install cattrs by default.

Airflow integration does not use deserialization, IIRC.
Regarding deserialize extra, this looks great. Could you remove Draft mark, please?

authors: JDarDagran dolfinus mgorsk1 mobuchowski


text:
Support to outsource Virtual Machines to OpenStack providers

Author Name: Carlo Daffara (Carlo Daffara)
Original Redmine Issue: 4302, https://dev.opennebula.org/issues/4302
Original Date: 2016-01-21

As much as I love OpenNebula, I had to witness several major IT services standardizing on OpenStack to create their own IaaS public services. It would greatly extend the range of available hybrid providers if OpenNebula provided a set of VMM_MAD remotes for openstack. A ruby library for low-level interfacing is already available ( https://github.com/ruby-openstack ) and is relatively similar to the AWS one.
A reasonable alternative would be a (not abandoned) multicloud driver, like jclouds or deltacloud.

hybrid drivers are no longer maintained, out of the scope of the project

authors: OpenNebulaProject tinova

repo: OpenNebula/one


text:
Speed script ?

hello ,
its possible to speed the script , maybe with the value speed walking ?
thanks

set the walk config value to a higher value.
Use at your own risk tho.

repo: OpenPoGo/OpenPoGoBot


text:
Added missing licence for jquery-i18n

Added recurser/jquery-i18n licence as per #1279

Thanks Owen!

authors: ostephens wetneb

repo: OpenRefine/OpenRefine


text:
Update wikibase-manifest-schema-v2.json

Remove comma so that the JSON is valid.
@wetneb: this seams required…

@wetneb: what do you think?
Regards, Antoine

repo: OpenRefine/wikibase-manifests


text:
RPI: compilation on latest raspbian image (debian stretch based) stops with an error

Hello,
compiling the latest snapshot 2.3.1 on a raspbian pi 3 with the latest raspbian image (based on debian stretch) ends with an error. With the last jessie-image from july 17 everything works fine.
Mavens output is attached: RobotArdu_failure_stretch.txt
Cheers corvus-albus
Infos about my enviroment:
$ java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) Client VM (build 25.65-b01, mixed mode)
--> you get the same error with 'openjdk version "1.8.0_141"'.
$ mvn --version
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T21:39:06+02:00)
Maven home: /opt/apache-maven-3.5.0
Java version: 1.8.0_65, vendor: Oracle Corporation
Java home: /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre
Default locale: de_DE, platform encoding: UTF-8
OS name: "linux", version: "4.9.35-v7+", arch: "arm", family: "unix"
$ uname -a
Linux ORServer-Corvi 4.9.41-v7+ #1023 SMP Tue Aug 8 16:00:15 BST 2017 armv7l GNU/Linux
$ cat /proc/device-tree/model
Raspberry Pi 3 Model B Rev 1.2
$ cat /etc/apt/sources.list.d/raspi.list
deb http://archive.raspberrypi.org/debian/ stretch main ui

Compilation for all robots (except C4EV3) works on stretch and buster.

authors: boonto corvus-albus

repo: OpenRoberta/openroberta-lab


text:
Anyone should be able to propose a meta-block

User Story
As a validator
I want to propose a meta-block
Because I want to avoid punishment.
Scenario
Given that auxiliary reaches the gas target
When I propose a meta-block on origin
Then I should be able to validate the seal.
Info
This story includes the proposal. Vote verification is another ticket.
Contracts:

OriginCore

There can be competing open proposals.

repo: OpenSTFoundation/mosaic-contracts


text:
How to translate strings during migration

We need a solution to translate strings during migrations. Proposal 1: Prompt for user input on command line (or gui). Proposal 2: Try to use default environment variables.
Use case is my draft of a custom welcome slide which should be created during initial app migration. Here I want to save a translated string into the title field.
Who can help?

I do not understand. What strings do you want to translate? Do you mean database migration?

Yes during database setup and migration. I changed the issue title some days ago.

So you want to translate values inside the Database?

We create models in the migration files, for example some custom slides. But there is no browser language at this moment. We have to set a language manually.

Not neccessary at the moment.

authors: normanjaeckel ostcar

repo: OpenSlides/OpenSlides


text:
Amendment sorting in motion overview incorrect

Amendmends are sorterd 1, 10, 11 ,12, 2, 3, 4, 5, 6, 7, 8, 9

OpenSlides version
Python version
Browser with version

OpenSlides 3 is sorting like you would expect, see the screenshot below.
what you describe is logic sorting. In OpenSlides 2, it's meant to sort this way.
In your settings.py chance the following add leading zeros to new motions:
MOTION_IDENTIFIER_MIN_DIGITS = 1
to
MOTION_IDENTIFIER_MIN_DIGITS = 3

Closed, due to inactivity and since this is not an issue

authors: cobii tsiegleauq


text:
Dev impl define contract

Component implementations can overload component contract to fullfill operational executions, eg: a Calculator component type can have a ScientificCalculatorImpl or SolarCalculatorImpl which can  define specific overload for the impl in a project or a project line.

Fix #188

authors: ELePors labordep

repo: OpenSmock/Molecule