text:
C#: Fallback to CoreCLR/MonoVM hosting APIs when hostfxr/NativeAOT fails

Some platforms don't support hostfxr but we can use the coreclr/monosgen library directly to initialize the runtime.
Android exports now use the android runtime identifier instead of linux-bionic, this removes the restrictions we previously had:

Adds support for all Android architectures (arm32, arm64, x32, and x64), previously only the 64-bit architectures were supported.
Loads System.Security.Cryptography.Native.Android (the .NET library that binds to the Android OS crypto functions).

Fixes https://github.com/godotengine/godot/issues/84559.

@raulsntos We'll need to create a separate mono flavor for the app module that includes the mono dependencies.
The addition of that flavor means we'll be able to guard mono specific logic in the java code by checking for the flavor:
if (BuildConfig.FLAVOR.equals("mono")) {
    // load mono library
}

This will also result in the creation of additional build templates specifically for the mono builds. @akien-mga we'll need to update the build tools logic for the mono build templates.

Could you explain what this means and how long to wait for corrections, and then review?

@Zamir7 As explained in the PR description, this adds support for 32-bit Android architectures and Android OS APIs (mainly crypto, see the linked issue).
This PR is marked as a draft, which means it's still a work-in-progress. It is tentatively planned for 4.4 (as you can see in the assigned milestone).

There's one more set of changes that'll be needed to support gradle builds.
export_plugin uses the assemble command for the gradle builds, so that logic will need to be updated to use assembleStandard on standard builds and assembleMono on mono builds.
The crypto jar file will also need to be copied to the build directory after it's downloaded. Alternatively if you know the path where it'll be downloaded, then app/build.gradlecan be updated to look in that path in addition to the path it's checking now.

The crypto jar file will also need to be copied to the build directory after it's downloaded. Alternatively if you know the path where it'll be downloaded, then app/build.gradle can be updated to look in that path in addition to the path it's checking now.

It should be copied because this file is retrieved when building the .NET project and it's stored in a temporary directory that will be deleted after the export. We can copy it as part of the ExportPlugin implementation, but I'd need a way to get the path to the gradle build directory as well as a way to detect if the export is part of a gradle build.
It looks like I can check if it's a gradle build with get_option("gradle_build/use_gradle_build"), and then use add_shared_object to copy the .jar file. It copies it to the res://android/libs/{BUILD_TYPE}/{ARCH} directory which I assume is enough for it to be included in the gradle build.

The crypto jar file will also need to be copied to the build directory after it's downloaded. Alternatively if you know the path where it'll be downloaded, then app/build.gradle can be updated to look in that path in addition to the path it's checking now.

It should be copied because this file is retrieved when building the .NET project and it's stored in a temporary directory that will be deleted after the export. We can copy it as part of the ExportPlugin implementation, but I'd need a way to get the path to the gradle build directory as well as a way to detect if the export is part of a gradle build.
It looks like I can check if it's a gradle build with get_option("gradle_build/use_gradle_build"), and then use add_shared_object to copy the .jar file. It copies it to the res://android/libs/{BUILD_TYPE}/{ARCH} directory which I assume is enough for it to be included in the gradle build.

Yes that should work!
EditorExportPlugin#_get_android_libraries(...) is another option as well, as that method only gets invoked by the gradle build.

EditorExportPlugin#_get_android_libraries(...) is another option as well, as that method only gets invoked by the gradle build.
This option however doesn't copy the jar file, so your current approach may be preferrable.

Yeah, I think the current approach is preferable, because the _get_android_libraries requires the jar to be inside res:// which won't be the case.
I did have to modify the app/build.gradle to include .jar files in subdirectories because the files added with add_shared_objects require specifying the architecture and it will include them under a directory with the name of the specified architecture.
https://github.com/godotengine/godot/blob/7b15f3b768eb088b5893579e70511fd21724e69a/platform/android/java/app/build.gradle#L45-L47

I am currently experiencing an issue with C # using HTTPS and SignalR in godot4.3 byte1, which involves all of my data related content. Is this bug related to SSL? Is there a fixed and usable version available now? Or other methods, thank you.

Is there something missing to merge this?

Thanks for the interest in this PR. This is a pretty big change for C# Android exports, so I'd like to be cautious. Testing is appreciated, here's a custom build of the editor you can use to test it:

:checkered_flag: Windows
:apple: macOS
:penguin: Linux

Editor x86_64
Editor universal
Editor x86_64

Editor x86_32

Editor x86_32

And the Android templates you'll need to use to export:

:robot: Android

Templates

Can verify this PR indeed fixed the linked issue on my device
https://github.com/user-attachments/assets/b9800224-607f-4c92-88ba-8432d443130e

Cryptography works as well.
https://github.com/user-attachments/assets/e698e886-76f7-46a0-9f3a-63113b677917

I also confirm that this build solves the problem with connecting to MySQl, thank you very much for the work done!

Thank you everyone for testing. If you could, please also share what was the host OS (the device from where you are exporting the game), the target architectures when exporting the APK, the architecture of the Android device (or emulator) where you run the built APK, and whether you used a gradle build.
Just want to confirm that it's tested in a wide variety of Android devices of different architectures (arm32, arm64, x82, x64). I'm also curious about the usability of the .NET Android workload, hence why I ask about the host OS. I expect Windows users to have no issues here, but I wonder about other OSs.

My device info:
Host OS: Harmony OS 4.2.0 / Android
Target Arch: aarch64 (arm64)
Supported ABI: arm64-v8a armeabi-v7a armeabi
CPU: kirin9000s
Android Version: Android 12 Snow Cone (API31)
GMS Support: false

Hit a crash here building my project with the binaries and template you shared @raulsntos:
Editor OS: OS X (M3/ARM)
Target device aarch64 (Google Pixel 6a)
Android 15
I do have native android and ios plugins enabled. The iOS one was built with 4.3 headers, would this cause problem?
Stack trace:
09-02 18:56:46.601 22286 22404 E godot   : USER ERROR: hostfxr_initialize_for_dotnet_command_line failed with code: -2147450745
09-02 18:56:46.601 22286 22404 E godot   :    at: initialize_hostfxr_self_contained (modules/mono/mono_gd/gd_mono.cpp:328)
09-02 18:56:46.601 22286 22404 E godot   : USER ERROR: Parameter "load_assembly_and_get_function_pointer" is null.
09-02 18:56:46.601 22286 22404 E godot   :    at: initialize_hostfxr_and_godot_plugins (modules/mono/mono_gd/gd_mono.cpp:395)
09-02 18:56:46.601 22286 22404 E godot   : USER ERROR: Parameter "godot_plugins_initialize" is null.
09-02 18:56:46.601 22286 22404 E godot   :    at: initialize (modules/mono/mono_gd/gd_mono.cpp:536)
09-02 18:56:46.644   549   549 W gralloc4: Unable to set buffer name SurfaceView[com.godot.example/com.godot.game.GodotApp]#1(BLAST Consumer)1: File name too long
09-02 18:56:46.647   549   549 W gralloc4: Unable to set buffer name SurfaceView[com.godot.example/com.godot.game.GodotApp]#1(BLAST Consumer)1: File name too long
09-02 18:56:46.648   549   549 W gralloc4: Unable to set buffer name SurfaceView[com.godot.example/com.godot.game.GodotApp]#1(BLAST Consumer)1: File name too long
09-02 18:56:46.657 22286 22404 F libc    : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 22404 (VkThread), pid 22286 (m.godot.example)

09-02 18:56:47.015 22473 22473 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-02 18:56:47.015 22473 22473 F DEBUG   : Build fingerprint: 'google/bluejay_beta/bluejay:15/AP31.240617.015/12207491:user/release-keys'
09-02 18:56:47.015 22473 22473 F DEBUG   : Revision: 'MP1.0'
09-02 18:56:47.015 22473 22473 F DEBUG   : ABI: 'arm64'
09-02 18:56:47.015 22473 22473 F DEBUG   : Timestamp: 2024-09-02 18:56:46.780680780-0300
09-02 18:56:47.015 22473 22473 F DEBUG   : Process uptime: 2s
09-02 18:56:47.015 22473 22473 F DEBUG   : Cmdline: com.godot.example
09-02 18:56:47.015 22473 22473 F DEBUG   : pid: 22286, tid: 22404, name: VkThread  >>> com.godot.example <<<
09-02 18:56:47.015 22473 22473 F DEBUG   : uid: 10379
09-02 18:56:47.015 22473 22473 F DEBUG   : tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
09-02 18:56:47.015 22473 22473 F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0000000000000000
09-02 18:56:47.015 22473 22473 F DEBUG   : Cause: null pointer dereference
09-02 18:56:47.015 22473 22473 F DEBUG   :     x0  b4000076213efb20  x1  00000072ece5d3d0  x2  0000000000000000  x3  0000000000000010
09-02 18:56:47.015 22473 22473 F DEBUG   :     x4  0000000000000037  x5  8000000000800000  x6  ff6160713fff7364  x7  7f7f7f7f7f7f7f7f
09-02 18:56:47.015 22473 22473 F DEBUG   :     x8  0000000000000000  x9  b4000074913b6170  x10 000000000000000a  x11 0000000000000000
09-02 18:56:47.015 22473 22473 F DEBUG   :     x12 b4000074715af7f0  x13 b4000074c178d020  x14 800260800000073b  x15 0000000000000000
09-02 18:56:47.015 22473 22473 F DEBUG   :     x16 00000073807162a0  x17 00000076d8478080  x18 00000072ec84c000  x19 b4000076213efb20
09-02 18:56:47.015 22473 22473 F DEBUG   :     x20 00000072ece5ea80  x21 00000072fa05a800  x22 0000000000000001  x23 b40000746143e1e8
09-02 18:56:47.015 22473 22473 F DEBUG   :     x24 b4000076213efb20  x25 b40000746143e1e8  x26 00000072ece5ea80  x27 0000000000000000
09-02 18:56:47.015 22473 22473 F DEBUG   :     x28 00000072fa06b000  x29 00000072ece5d400
09-02 18:56:47.015 22473 22473 F DEBUG   :     lr  00000072f74b7b74  sp  00000072ece5d3b0  pc  0000000000000000  pst 0000000060001000
09-02 18:56:47.015 22473 22473 F DEBUG   : 29 total frames
09-02 18:56:47.015 22473 22473 F DEBUG   : backtrace:
09-02 18:56:47.015 22473 22473 F DEBUG   :       #00 pc 0000000000000000  <unknown>
09-02 18:56:47.015 22473 22473 F DEBUG   :       #01 pc 0000000001458b70  /data/app/~~2zH5GIXjYyB7gNeoXrW8Mw==/com.godot.example-S3lOFmMDdHq6Tv8w5K95DQ==/base.apk!libgodot_android.so (offset 0xa80000)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #02 pc 0000000001459e6c  /data/app/~~2zH5GIXjYyB7gNeoXrW8Mw==/com.godot.example-S3lOFmMDdHq6Tv8w5K95DQ==/base.apk!libgodot_android.so (offset 0xa80000)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #03 pc 0000000003853650  /data/app/~~2zH5GIXjYyB7gNeoXrW8Mw==/com.godot.example-S3lOFmMDdHq6Tv8w5K95DQ==/base.apk!libgodot_android.so (offset 0xa80000)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #04 pc 0000000003853ec8  /data/app/~~2zH5GIXjYyB7gNeoXrW8Mw==/com.godot.example-S3lOFmMDdHq6Tv8w5K95DQ==/base.apk!libgodot_android.so (offset 0xa80000)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #05 pc 0000000003855418  /data/app/~~2zH5GIXjYyB7gNeoXrW8Mw==/com.godot.example-S3lOFmMDdHq6Tv8w5K95DQ==/base.apk!libgodot_android.so (offset 0xa80000)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #06 pc 00000000038555b4  /data/app/~~2zH5GIXjYyB7gNeoXrW8Mw==/com.godot.example-S3lOFmMDdHq6Tv8w5K95DQ==/base.apk!libgodot_android.so (offset 0xa80000)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #07 pc 0000000000f23640  /data/app/~~2zH5GIXjYyB7gNeoXrW8Mw==/com.godot.example-S3lOFmMDdHq6Tv8w5K95DQ==/base.apk!libgodot_android.so (offset 0xa80000)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #08 pc 0000000000eec2bc  /data/app/~~2zH5GIXjYyB7gNeoXrW8Mw==/com.godot.example-S3lOFmMDdHq6Tv8w5K95DQ==/base.apk!libgodot_android.so (offset 0xa80000) (Java_org_godotengine_godot_GodotLib_step+172)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #09 pc 0000000000378f70  /apex/com.android.art/lib64/libart.so (art_quick_generic_jni_trampoline+144) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #10 pc 0000000000362a40  /apex/com.android.art/lib64/libart.so (art_quick_invoke_static_stub+640) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #11 pc 000000000035bd94  /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, bool, art::JValue*)+2048) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #12 pc 000000000076df08  /apex/com.android.art/lib64/libart.so (void art::interpreter::ExecuteSwitchImplCpp<false>(art::interpreter::SwitchImplContext*)+12208) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #13 pc 000000000037b5d8  /apex/com.android.art/lib64/libart.so (ExecuteSwitchImplAsm+8) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #14 pc 0000000000110a44  [anon:dalvik-classes5.dex extracted in memory from /data/app/~~2zH5GIXjYyB7gNeoXrW8Mw==/com.godot.example-S3lOFmMDdHq6Tv8w5K95DQ==/base.apk] (org.godotengine.godot.vulkan.VkRenderer.onVkDrawFrame+0)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #15 pc 000000000034e21c  /apex/com.android.art/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.__uniq.112435418011751916792819755956732575238.llvm.2845697060370838518)+428) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #16 pc 000000000035c5b0  /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, bool, art::JValue*)+4124) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #17 pc 000000000076df08  /apex/com.android.art/lib64/libart.so (void art::interpreter::ExecuteSwitchImplCpp<false>(art::interpreter::SwitchImplContext*)+12208) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #18 pc 000000000037b5d8  /apex/com.android.art/lib64/libart.so (ExecuteSwitchImplAsm+8) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #19 pc 00000000001111f4  [anon:dalvik-classes5.dex extracted in memory from /data/app/~~2zH5GIXjYyB7gNeoXrW8Mw==/com.godot.example-S3lOFmMDdHq6Tv8w5K95DQ==/base.apk] (org.godotengine.godot.vulkan.VkThread.run+0)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #20 pc 000000000034d5a8  /apex/com.android.art/lib64/libart.so (artQuickToInterpreterBridge+1932) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #21 pc 0000000000379098  /apex/com.android.art/lib64/libart.so (art_quick_to_interpreter_bridge+88) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #22 pc 0000000000362774  /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #23 pc 000000000034def0  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+132) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #24 pc 0000000000943da8  /apex/com.android.art/lib64/libart.so (art::detail::ShortyTraits<(char)86>::Type art::ArtMethod::InvokeInstance<(char)86>(art::Thread*, art::ObjPtr<art::mirror::Object>, art::detail::ShortyTraits<>::Type...)+60) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #25 pc 000000000063eb68  /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1344) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #26 pc 000000000063e618  /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallbackWithUffdGc(void*)+8) (BuildId: 7300f6c36a74cc43266451736b120528)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #27 pc 000000000006f718  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+200) (BuildId: a923caae2e75861e7a06a23f740068a5)
09-02 18:56:47.015 22473 22473 F DEBUG   :       #28 pc 0000000000060e00  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: a923caae2e75861e7a06a23f740068a5)

@felipejfc Thanks for testing. From that stack trace, it looks like it's trying to use hostfxr to initialize the .NET runtime which shouldn't be possible since the Android export shouldn't contain libhostfxr.so. Can you share the csproj and the APK?
I'd like to see the csproj so I can check if there's anything that could affect the publish process. For example, something like UseMonoRuntime or PublishAot.
And I'd like to see the APK to check what native libraries were included in the lib directory, specifically I'd expect to see libmonosgen-2.0.so there. You can check yourself opening the APK in Android Studio.

I do have native android and ios plugins enabled. The iOS one was built with 4.3 headers, would this cause problem?

I don't think so. It looks like the error is caused by failing to initialize the .NET runtime, not sure what may be causing it. I'd like to see what native libraries are included in the APK. And, if libmonosgen-2.0 is missing, we'd need to investigate why that's the case.

@felipejfc Thanks for testing. From that stack trace, it looks like it's trying to use hostfxr to initialize the .NET runtime which shouldn't be possible since the Android export shouldn't contain libhostfxr.so. Can you share the csproj and the APK?
I'd like to see the csproj so I can check if there's anything that could affect the publish process. For example, something like UseMonoRuntime or PublishAot.
And I'd like to see the APK to check what native libraries were included in the lib directory, specifically I'd expect to see libmonosgen-2.0.so there. You can check yourself opening the APK in Android Studio.

I do have native android and ios plugins enabled. The iOS one was built with 4.3 headers, would this cause problem?

I don't think so. It looks like the error is caused by failing to initialize the .NET runtime, not sure what may be causing it. I'd like to see what native libraries are included in the APK. And, if libmonosgen-2.0 is missing, we'd need to investigate why that's the case.

Sadly I can't share the whole files since this is a private company project, however, I can provide all info you asked for:
<Project Sdk="Godot.NET.Sdk/4.3.0">
  <PropertyGroup>
    <EnableAot>false</EnableAot>
    <EnableAot Condition="'$(GodotTargetPlatform)' == 'ios'">true</EnableAot>
    <PublishAotUsingRuntimePack>$(EnableAot)</PublishAotUsingRuntimePack>
    <PublishAot Condition=" ('$(NETCoreSdkRuntimeIdentifier)' == 'linux-x64' or $(NETCoreSdkRuntimeIdentifier.StartsWith('osx')) ) and ('$(GodotTargetPlatform)' == 'ios' or '$(GodotTargetPlatform)' == 'android') and '$(EnableAot)' == 'true' ">true</PublishAot>
    <TargetFramework>net8.0</TargetFramework>
    <EnableDynamicLoading>true</EnableDynamicLoading>
    <JsonSerializerIsReflectionEnabledByDefault>false</JsonSerializerIsReflectionEnabledByDefault>
    <RootNamespace>GodotExample</RootNamespace>
  </PropertyGroup>
  <ItemGroup>
    <!-- Root the assemblies to avoid trimming. -->
    <PackageReference Include="Google.Protobuf" Version="3.27.3" />
    <PackageReference Include="UniTask" Version="2.5.5" />
    <TrimmerRootAssembly Include="GodotSharp" />
    <TrimmerRootAssembly Include="$(TargetName)" />
    <!-- Add reference to the source generator project -->
    <ProjectReference Include="sourcegenerator/SourceGenerator.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />
    <ProjectReference Include="./sharedtypes/sharedtypes.csproj" />
  </ItemGroup>
  <ItemGroup>
    <Compile Remove="sourcegenerator/**/*.cs" />
    <Compile Remove="sharedtypes/**/*.cs" />
  </ItemGroup>
  <Target Name="WarnIfAndroidWithoutAot" BeforeTargets="Build">
    <Warning Text="AOT is not being used in the generated APK. AOT for Android is only supported when building from macOS or Linux. Current Host RuntimeIdentifier: $(NETCoreSdkRuntimeIdentifier)" Condition="'$(GodotTargetPlatform)' == 'android' and '$(EnableAot)' == 'true' and !('$(NETCoreSdkRuntimeIdentifier)' == 'linux-x64' or $(NETCoreSdkRuntimeIdentifier.StartsWith('osx')))" />
  </Target>
  <Import Project="android.targets" />
</Project>

This is the content of android.targets:
<Project>
  <PropertyGroup>
    <_NdkSysrootAbi Condition=" '$(RuntimeIdentifier)' == 'linux-bionic-arm64' or '$(RuntimeIdentifier)' == 'android-arm64' ">aarch64-linux-android</_NdkSysrootAbi>
    <_NdkClangPrefix Condition=" '$(RuntimeIdentifier)' == 'linux-bionic-arm64' or '$(RuntimeIdentifier)' == 'android-arm64' ">aarch64-linux-android21-</_NdkClangPrefix>
    <_NdkPrebuiltAbi Condition=" $(NETCoreSdkRuntimeIdentifier.StartsWith('osx')) ">darwin-x86_64</_NdkPrebuiltAbi>
    <_NdkPrebuiltAbi Condition=" '$(NETCoreSdkRuntimeIdentifier)' == 'linux-x64' ">linux-x86_64</_NdkPrebuiltAbi>
    <_NdkPrebuiltAbi Condition=" '$(NETCoreSdkRuntimeIdentifier)' == 'win-x64' ">windows-x86_64</_NdkPrebuiltAbi>
    <_NdkSysrootDir>$(ANDROID_NDK_HOME)/toolchains/llvm/prebuilt/$(_NdkPrebuiltAbi)/sysroot/usr/lib/$(_NdkSysrootAbi)</_NdkSysrootDir>
    <_NdkBinDir>$(ANDROID_NDK_HOME)/toolchains/llvm/prebuilt/$(_NdkPrebuiltAbi)/bin</_NdkBinDir>
  </PropertyGroup>

  <PropertyGroup Condition="'$(EnableAOT)' == 'true' and ($(RuntimeIdentifier.StartsWith('linux-bionic')) Or $(RuntimeIdentifier.StartsWith('android-arm64')))">
    <CppCompilerAndLinker>$(_NdkBinDir)/$(_NdkClangPrefix)clang++</CppCompilerAndLinker>
    <ObjCopyName>$(_NdkBinDir)/llvm-objcopy</ObjCopyName>
  </PropertyGroup>

  <ItemGroup Condition="'$(EnableAOT)' == 'true' and ($(RuntimeIdentifier.StartsWith('linux-bionic')) Or $(RuntimeIdentifier.StartsWith('android-arm64')))">
    <LinkerArg Include="-Wl,--undefined-version" />
  </ItemGroup>

  <Target Name="_ValidateEnvironment"
      BeforeTargets="Build"
      Condition="'$(EnableAOT)' == 'true' and ($(RuntimeIdentifier.StartsWith('linux-bionic')) or $(RuntimeIdentifier.StartsWith('android-arm64'))) and ('$(NETCoreSdkRuntimeIdentifier)' == 'linux-x64' or $(NETCoreSdkRuntimeIdentifier.StartsWith('osx'))) ">
    <Error
        Condition=" '$(ANDROID_NDK_HOME)' == '' Or !Exists($(ANDROID_NDK_HOME)) "
        Text="Set the %24ANDROID_NDK_HOME environment variable to the path of the Android NDK."
     />
    <Error
        Condition=" !Exists($(_NdkSysrootDir))"
        Text="NDK 'sysroot' dir `$(_NdkSysrootDir)` does not exist. You're on your own."
    />
  </Target>
 </Project>

This should only be used if NativeAOT + Android build which I'm not setting right now as you can see, only for iOS, but it's there as a placeholder once I want to start building android with it.
As for the libs, seems to be there:

More logging from debugging with android studio:
  I  Godot Engine v4.4.android-monovm.mono.custom_build.95c5436e3 - https://godotengine.org
2024-09-02 20:29:19.417 30752-30845 godot                   com.example.godot                    I  Vulkan 1.3.274 - Forward Mobile - Using Device #0: ARM - Mali-G78
2024-09-02 20:29:19.420 30752-30845 gralloc4                com.example.godot                    E  ERROR: Format allocation info not found for format: 3b
2024-09-02 20:29:19.420 30752-30845 gralloc4                com.example.godot                    E  ERROR: Format allocation info not found for format: 0
2024-09-02 20:29:19.421 30752-30845 gralloc4                com.example.godot                    E  Invalid base format! req_base_format = (<unrecognized format> 0x0), req_format = (<unrecognized format> 0x3b), type = 0x0
2024-09-02 20:29:19.421 30752-30845 gralloc4                com.example.godot                    E  ERROR: Unrecognized and/or unsupported format (<unrecognized format> 0x3b) and usage (CPU_READ_NEVER|CPU_WRITE_NEVER|GPU_TEXTURE|GPU_RENDER_TARGET|COMPOSER_OVERLAY 0xb00)
2024-09-02 20:29:19.421 30752-30845 gralloc4                com.example.godot                    E  ERROR: Format allocation info not found for format: 3b
2024-09-02 20:29:19.421 30752-30845 gralloc4                com.example.godot                    E  ERROR: Format allocation info not found for format: 0
2024-09-02 20:29:19.421 30752-30845 gralloc4                com.example.godot                    E  Invalid base format! req_base_format = (<unrecognized format> 0x0), req_format = (<unrecognized format> 0x3b), type = 0x0
2024-09-02 20:29:19.421 30752-30845 gralloc4                com.example.godot                    E  ERROR: Unrecognized and/or unsupported format (<unrecognized format> 0x3b) and usage (CPU_READ_NEVER|CPU_WRITE_NEVER|GPU_TEXTURE|GPU_RENDER_TARGET|COMPOSER_OVERLAY 0xb00)
2024-09-02 20:29:19.515 30752-30845 m.example.godot         com.example.godot                    D  PlayerBase::PlayerBase()
2024-09-02 20:29:19.519 30752-30845 m.example.godot         com.example.godot                    D  TrackPlayerBase::TrackPlayerBase()
2024-09-02 20:29:19.519 30752-30845 libOpenSLES             com.example.godot                    I  Emulating old channel mask behavior (ignoring positional mask 0x3, using default mask 0x3 based on channel count of 2)
2024-09-02 20:29:19.532 30752-30845 godot                   com.example.godot                    I
2024-09-02 20:29:19.656 30752-30752 VkThread                com.example.godot                    W  type=1400 audit(0.0:24003): avc:  granted  { execute } for  path="/data/data/com.example.godot/cache/data_RoyalMetagameGodot_android_arm64/libhostfxr.so" dev="dm-56" ino=40929 scontext=u:r:untrusted_app:s0:c123,c257,c512,c768 tcontext=u:object_r:app_data_file:s0:c123,c257,c512,c768 tclass=file app=com.example.godot
2024-09-02 20:29:19.656 30752-30752 VkThread                com.example.godot                    W  type=1400 audit(0.0:24004): avc:  granted  { execute } for  path="/data/data/com.example.godot/cache/data_RoyalMetagameGodot_android_arm64/libhostpolicy.so" dev="dm-56" ino=37304 scontext=u:r:untrusted_app:s0:c123,c257,c512,c768 tcontext=u:object_r:app_data_file:s0:c123,c257,c512,c768 tclass=file app=com.example.godot
2024-09-02 20:29:19.664 30752-30845 godot                   com.example.godot                    E  USER ERROR: hostfxr_initialize_for_dotnet_command_line failed with code: -2147450745
2024-09-02 20:29:19.664 30752-30845 godot                   com.example.godot                    E     at: initialize_hostfxr_self_contained (modules/mono/mono_gd/gd_mono.cpp:328)
2024-09-02 20:29:19.664 30752-30845 godot                   com.example.godot                    E  USER ERROR: Parameter "load_assembly_and_get_function_pointer" is null.
2024-09-02 20:29:19.664 30752-30845 godot                   com.example.godot                    E     at: initialize_hostfxr_and_godot_plugins (modules/mono/mono_gd/gd_mono.cpp:395)
2024-09-02 20:29:19.664 30752-30845 godot                   com.example.godot                    E  USER ERROR: Parameter "godot_plugins_initialize" is null.
2024-09-02 20:29:19.664 30752-30845 godot                   com.example.godot                    E     at: initialize (modules/mono/mono_gd/gd_mono.cpp:536)
2024-09-02 20:29:19.690 30752-30845 libc                    com.example.godot                    A  Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 30845 (VkThread), pid 30752 (m.example.godot)
2024-09-02 20:29:19.722 30752-30888 ProfileInstaller        com.example.godot                    D  Installing profile for com.example.godot
2024-09-02 20:29:19.789 30752-30752 InteractionJankMonitor  com.example.godot                    W  Initializing without READ_DEVICE_CONFIG permission. enabled=false, interval=1, missedFrameThreshold=3, frameTimeThreshold=64, package=com.example.godot

Thanks for sharing, I don't see anything in the csproj or android.targets files that seems problematic.
From the screenshot you shared of the APK, I'd say the publish process was successful. It's weird that it's trying to use hostfxr since there's no libhostfxr.so.
By any chance have you exported this game before with a previous version of Godot? Since Godot extracts the contents of the APK to a cache directory, it's possible that previously cached files are still there causing issues.
It looks like the cache path is /data/data/com.example.godot/cache/data_RoyalMetagameGodot_android_arm64, could you try to delete that directory before executing the game again? Alternatively, changing the Android application ID before exporting may also work.

Spot on! Deleted cache and uninstalled the app, installed it again and it works! We will do good then adding this disclaimer that the application needs to be installed with a fresh cache if the player had a version generated with an old godot build installed

We will do good then adding this disclaimer that the application needs to be installed with a fresh cache if the player had a version generated with an old godot build installed

We'll fix it so it automatically clears those cached files if the extracted .dotnet-publish-manifest file is outdated, so the disclaimer won't be needed.

When building for armv7 I got the following error Unable to find package Microsoft.NETCore.App.Runtime.linux-bionic-arm in some csprojs:
Which was fixed by https://github.com/dotnet/android/issues/8170#issuecomment-1623845666

Thanks a lot, that's really interesting. As mentioned in that issue, I suspect your referenced projects (i.e.: ./sharedtypes/sharedtypes.csproj) are not setting UseMonoRuntime=true.
We set UseMonoRuntime=true in the Godot.NET.Sdk, and I assume your referenced projects are using Microsoft.NET.Sdk. Perhaps we can add the workaround you linked in the Godot.NET.Sdk, as a temporary measure until the issue is fixed upstream.

Thanks a lot, that's really interesting. As mentioned in that issue, I suspect your referenced projects (i.e.: ./sharedtypes/sharedtypes.csproj) are not setting UseMonoRuntime=true.

They are not! Should I reference it?

We'll fix it so it automatically clears those cached files if the extracted .dotnet-publish-manifest file is outdated, so the disclaimer won't be needed.

I want to mention that the Windows Single File export feature, which involves embedding script files, is currently encountering a similar issue. Specifically, the single-file executable stops working after switching to AOT because the outdated managed assemblies inside the cache directory are loaded again upon launching the new AOT executable.

They are not! Should I reference it?

@felipejfc If the workaround mentioned in the issue you linked doesn't work, you could try doing that. Ideally you shouldn't have to do anything manually.

I want to mention that the Windows Single File export feature, which involves embedding script files, is currently encountering a similar issue.

@Delsin-Yu Yeah, this is an existing issue. It's reported in https://github.com/godotengine/godot/issues/96299.

@raulsntos Tested on an out Motorola ARMv7 device with Adreno GPU
1 - Build crashed with Mobile rendered (because of vulkan I assume)
godot E  USER ERROR: .NET: Failed to get GodotPlugins initialization function pointer
      E     at: initialize_coreclr_and_godot_plugins (modules/mono/mono_gd/gd_mono.cpp:485)
      E  USER ERROR: .NET: Failed to load hostfxr
      E     at: initialize (modules/mono/mono_gd/gd_mono.cpp:549)
      E  USER ERROR: BUG: Unreferenced static string to 0: ShaderCompilation
      E     at: unref (core/string/string_name.cpp:127)
libc  F  FORTIFY: pthread_mutex_lock called on a destroyed mutex (0x81bb9e74)`

2 - When built with OpenGL the game worked fine!
Idk if the crash has to do with godot or with vulkan driver incompatibility but I would guess the later

I would not expect to see the Failed to load hostfxr error, it sounds like there may be a libhostfxr.so in the cache again^1.

Rebased and made new custom builds to test, the cached data should no longer be an issue since https://github.com/godotengine/godot/pull/96301 was merged:

:checkered_flag: Windows
:apple: macOS
:penguin: Linux

Editor x86_64
Editor universal
Editor x86_64

Editor x86_32

Editor x86_32

And the Android templates you'll need to use to export:

:robot: Android

Templates

I’m curious, after the PR is merged, can I still use this change #86791 and export it to Android via aot?

NativeAOT should not be affected by this PR. If your csproj contains <PublishAot>true</PublishAot>, then it won't use the Mono runtime, the libmonosgen-2.0.so library won't be in the exported project so the changes in this PR will not take place.
Keep in mind NativeAOT for Android is still experimental upstream, and will likely require more changes to make it work properly but that's outside of the scope of this PR which only focuses on enabling the Mono runtime.

https://github.com/dotnet/runtime/issues/106748.

Any chance we backport this to 4.3? Using native binding for handling http calls in android is a hassle

I don't think so, this is a pretty big change to C# Android exports that builds on top of a number of other changes to the Android platform that I don't expect to see backported to 4.3 either.
Issues with crypto functions not being available in 4.3 can be worked around by avoiding crypto APIs from the BCL or thirdparty C# libraries, and instead relying on the crypto APIs provided by Godot (i.e.: Use Godot.HttpClient instead of System.Net.Http.HttpClient).
Alternatively, you can include the required native libraries (libssl.so and libcrypto.so) in the exported APK and that should work too but I reckon it's even more of a hassle to do this.

Needs rebase after #96967.

Thanks! Amazing work :tada:


text:
Fix potential infinite loop when using global menu

It's always checking the same iterator 🤣
I think it's a copy paste error.

Thanks!

source: gharchive/pull-request

authors: akien-mga timothyqiu

repo: godotengine/godot

license: MIT

license_type: permissive

license_source: github-api


text:
Add support for custom sections in plugin.cfg files

Implemented functionality allowing plugins to define custom sections in their plugin.cfg files. Plugin developers can now add and manage custom configuration settings directly in plugin.cfg. Enhanced the Project Settings window to display and edit these custom sections and settings.
Screenshots with an example plugin.cfg
name="Skibidi Plugin"
author="Murat"
description=""
version="1.0.0"
script="skibidi.gd"

[sodas]
coke=true
pepsi=false
monster=true
redbull=true

[sky]
sky_color=Color(135, 206, 235, 255)
heaven_gate_position=Vector2(123, 456)

[misc]
unicorns=["unicorn1", "unicorn2"]
ants={"ant1": {"name": "notsoant"}}
gravity=420

Project Settings

Accessing values with get_plugin_settings()
@tool
extends EditorPlugin

func _enter_tree() -> void:
	print("Loaded Skibidi Plugin")
	print(get_plugin_settings())
	pass

func _exit_tree() -> void:
	pass

Changing values

Thank you for your contribution, and welcome to contributing?
Is there a proposal for this feature open? Feature pull requests should be associated to a feature proposal to justify the need for implementing the feature in Godot core. If there isn't already one please open a proposal on the Godot proposals repository (and link to this pull request in the proposal's body).
See here for general instructions and instructions for our PR workflow

authors: AThousandShips Murateba


text:
Autostart on Centos 6

I installed Gogs on my Centos 6 machine using the "binary" guide.
I created a git user and did all the steps. I also added the gogs startup script (from https://github.com/gogits/gogs/blob/master/scripts/init/centos/gogs) to my /etc/init.d folder and added the startup script to my startup sequence using chkconfig --add gogs. The script is listed as S85gogs in:

/etc/rc2.d
/etc/rc3.d
/etc/rc4.d
/etc/rc5.d

I tested my configuration using sudo service gogs start, which starts the gogs service properly and it is reachable through my proxy in apache (yay!).
After that I rebooted my machine to see if it started automatically, but it is not auto starting. The boot.log shows that gogs starts correctly:
Starting Gogs Go Git Service: ^[[60G[^[[0;32m  OK  ^[[0;39m]
But the gogs service is still unavailable. Also a sudo service gogs status shows that my gogs service is stopped.
I have the feeling this is a permissions problem, but I can't seem to find out why. My gogs permissions are -rwxr-xr-x.  1 root root  1774 Mar 26 11:33 gogs. These are the same as the other services running at my system.
One minor detail, when my git user tries to start the service, I get the following message, not sure this is related to the problem or just because the user isn't tied to any group.
$ service gogs start
Starting Gogs Go Git Service: runuser: cannot set groups: Operation not permitted
                                                           [FAILED]

Hi, I'm not sure this is the reason Gogs fails to start :+1:

Starting Gogs Go Git Service: runuser: cannot set groups: Operation not permitted

Maybe you need to use sudo service gogs start for this.
For reboot, do you have any log from log/gogs.log? There should be some error message if Gogs itself refuse to start.

Also from some search, it might be a SELinux issue: http://unix.stackexchange.com/questions/80066/postgresql-service-start-issues

Yup, sudo service gogs start does work. I mentioned this in my original post.
The logs in the /log folder do show an error indeed. It seems that my MySQL server isn't started before my gogs service is started. There is a mysqld script in init.d, but it doesn't have an on in my chkconfig --list. Maybe this is because it is started by Plesk (which is installed on my server too).
Damn, I wish I knew more about this on Centos.

Right, does init has something like require a service to start before this one? Like systemd.

BTW, I'm using CentOS 7 with systemd and it works well for weeks so far.

Upgrading is something I had in mind for a couple of weeks now, but I am hosting a couple of sites and I do need Plesk to keep everything organized. Not sure an upgrade will go so well.
In other news, the gogs init.d script does have this:
### BEGIN INIT INFO
# Provides:          gogs
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start gogs at boot time.
# Description:       Control gogs.
### END INIT INFO

I wonder if I can do something with the "Required-Start" from this.

Required-Start looks the right way to try.
PS: my point isn't CenOS 7, but systemd, if you would consider to switch to it. I'm not pro on this kind of thing but at least I got working solution :eyes:

Ah yes, well I will try to fiddle with the Required-Start.
As I read it, systemd is only for Centos 7?

Oops, my bad... that's so sad!

No problem, it seems solved now. I downgraded the priority from 85 to 99 and that fixed it. My Plesk control panel also had the priority 85 and that is now started before gogs.
Awesome! Thanks for the pointers!

No problem, my pleasure!

source: gharchive/issue

repo: gogits/gogs


text:
Support of PGP signed commits and tags

Just showing, who signed them will be great

+1. Would be awesome to have this in Gogs now that it's been deployed as a GitHub feature!

+1

Agree that this should be a thing.

I would very much so enjoy this feature, as well!

I'm claiming this :)

I am adding my "voice" to this request. I love Gogs!

I would love this too!

I'd really wanna do this in the frontend only, since Gogs as a service don't really care. Anyone against that?
Note to Self: https://github.com/openpgpjs/openpgpjs

It will be an opportunity to implement this feature?

authors: Cysioland Fastidious Jitsusama bashrc bkcsoft ikaros qdbdbp tuxxy verandaguy


text:
Docker gogs web/ssh does not restart after reboot

I am using a fresh install from docker gogs/gogs on Linux. Initially, gogs works fine. But when I reboot the host, the dockers do relaunch but do not serve the web interface :(
curl http://127.0.0.1:10080
curl: (7) Failed to connect to 127.0.0.1 port 10080: Connection refused

This is how I started gogs. Note the restart policy + I am publishing ports 10022 and 10080.
docker run --restart=always --name gogs-db -e MYSQL_DATABASE=gogs -e MYSQL_ROOT_PASSWORD=MYSECRET -d mysql:5.6.27
docker run --restart=always -d --name gogs-cuckoo --link gogs-db:mysql --publish 10022:22 --publish 10080:3000 -v /var/gogs:/data gogs/gogs

I continued the installation without any problems, and then managed to log on to gogs and work with it without any problem. The web interface is expected to work on http://MYIP:10080, and that worked fine, and I created repositories etc.
However, when I reboot the host, the 2 docker containers do relaunch thanks to the restart policy
axelle@cuckoo:~$ docker ps
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                                                    NAMES
05d957d48fd6        gogs/gogs            "docker/start.sh /bin"   2 hours ago         Up 2 hours          0.0.0.0:10022->22/tcp, 0.0.0.0:10080->3000/tcp   gogs-cuckoo
aa9ee4739825        mysql:5.6.27         "/entrypoint.sh mysql"   6 hours ago         Up 2 hours          3306/tcp                                                 gogs-db

but they do not serve the HTTP (nor SSH).
$ curl http://127.0.0.1:10080
curl: (7) Failed to connect to 127.0.0.1 port 10080: Connection refused

Though gogs seems to be listening on 10080 correctly:
$ docker logs gogs-cuckoo 2016/04/29 12:58:18 [T] Custom path: /data/gogs 2016/04/29 12:58:18 [T] Log path: /app/gogs/log 2016/04/29 12:58:18 [I] Gogs: Go Git Service 0.9.22.0425 2016/04/29 12:58:18 [I] Log Mode: File(Info) 2016/04/29 12:58:18 [I] Cache Service Enabled 2016/04/29 12:58:18 [I] Session Service Enabled 2016/04/29 12:58:18 [I] Git Version: 2.6.6 2016/04/29 12:58:23 [I] SQLite3 Supported 2016/04/29 12:58:23 [I] Run Mode: Production 2016/04/29 12:58:23 [T] Doing: CheckRepoStats 2016/04/29 12:58:23 [I] Listen: http://0.0.0.0:10080 2016/04/29 13:58:23 [T] Doing: MirrorUpdate 2016/04/29 14:58:23 [T] Doing: MirrorUpdate

There's your erfor, it should listen on 3000, not 10080

@cryptax was this resolved?

@bkcsoft No, that does not solve it.
The curl http://127.0.0.1:10080 I am trying is not inside the docker, but outside. So, it should not be port 3000, but port 10080. Indeed, the docker is publishing port 3000 to 10080 in my configuration (see --publish 10080:3000).
Nevertheless, I tried it, and I confirm it does not work:
$ curl http://127.0.0.1:3000
curl: (7) Failed to connect to 127.0.0.1 port 3000: Connection refused
axelle@cuckoo:~$ curl http://127.0.0.1:10080
curl: (56) Recv failure: Connection reset by peer

And the port 10080 is shown as listening:
$ netstat -tulpn | grep 10080
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::10080                :::*                    LISTEN      -

I currently have no particular error in the docker logs:
2016/04/29 15:14:08 [I] Listen: http://0.0.0.0:10080
...
2016/05/02 06:14:08 [T] Doing: MirrorUpdate
2016/05/02 07:14:08 [T] Doing: MirrorUpdate
2016/05/02 08:14:08 [T] Doing: MirrorUpdate

There is something strange, though, when I get in the container, it seems the web is listening on 10080. I would have expected 3000.
axelle@cuckoo:~$ docker exec -it gogs-cuckoo bash
bash-4.3# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      32/socat
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      33/sshd
tcp        0      0 :::10080                :::*                    LISTEN      -
tcp        0      0 :::22                   :::*                    LISTEN      33/sshd

As you can see here:

2016/04/29 15:14:08 [I] Listen: http://0.0.0.0:10080

Gogs is listening on 10080 not 3000 as it should :smile:

@bkcsoft well, no the gogs container does not respond neither on 3000 nor on 10080.
$ curl http://127.0.0.1:3000
curl: (7) Failed to connect to 127.0.0.1 port 3000: Connection refused
$ curl http://127.0.0.1:10080
curl: (56) Recv failure: Connection reset by peer

Ok - looks like I misread the install guidelines.
Everything is fine.
For those having the same issue as me:

domain: ensure that is the IP address of your docker host
SSH port: that is the forwarded ssh port. So if you forward 22 to 10022, it's 10022.
HTTP port: my error, here, you must enter the HTTP in the container. So it's 3000. Even if you forward it to 10080 ;)
Application URL: this is a combination of domain + forwarded HTTP port (I got this one wrong too). So, if you're forwarding 3000 to 10080, then it's http://domain:10080.

The issue can be closed, thanks!

authors: bkcsoft cryptax


text:
Unable to modify picture

Gogs version (or commit ref): 0.9.128.0131
Git version: 1.7.1
Operating system: centos 6.8 64bit
Database (use [x]):

[x] SQLite

Can you reproduce the bug at https://try.gogs.io:

[x] Yes

Description
Unable to modify picture This includes：The person's image | organization logo

Hi, please read post template very carefully, I don't know what you are saying.

I have the same problem, in https://my.site/org/orgname/settings:

Upload a new PNG-image for an organisation → Result: Still the initial default profile image is shown.
Remove the profile image → This works, a 45°-turned G is shown, nmot the logo
Upload a new PNG-image for an organisation → Result: The initial default profile image is restored.

Proxy settings look good, they are:
server {
  listen 443;
  server_name mrw.sh;
  ssl on;
  ssl_certificate /etc/letsencrypt/live/mrw.sh/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/mrw.sh/privkey.pem;
  location / {
    include proxy.conf;
    if ($request_method ~ ^COPY$) {
      rewrite /(.*) /$1 break;
    }
    #if ( $host != 'mrw.sh' ) {
    #  rewrite ^/(.*)$ $scheme://mrw.sh/$1 permanent;
    #}
    proxy_cookie_domain universum mrw.sh;
    proxy_cookie_path / /;
    proxy_pass http://universum:8012/;
    proxy_redirect http://universum:8012/ /;
    proxy_redirect http://universum:8012/ $scheme://mrw.sh/;
    proxy_redirect / $scheme://mrw.sh/;
    proxy_redirect / /;
    proxy_redirect / $scheme://mrw.sh/;
    subs_filter "http://universum:8012" "$scheme://mrw.sh";
    subs_filter "universum:8012" "mrw.sh";
  }
}

With proxy.conf:
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
set $ssl off;
if ($scheme = https) {
    set $ssl on;
}
proxy_set_header X-Forwarded-Ssl $ssl;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Accept-Encoding "";
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Original-Request $request_uri;
proxy_pass_request_headers on;
#proxy_cache off;
#proxy_buffering off;
client_max_body_size 4096m;
client_body_buffer_size 128k;
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
proxy_buffers 32 4k;
subs_filter_types text/css text/javascript text/xml;
set $fixed_destination $http_destination;
if ( $http_destination ~* ^https(.*)$ ) {
  set $fixed_destination http$1;
}
proxy_set_header Destination $fixed_destination;

So, I don't think it's the reverse-proxy?

Not sure if it's the problem with reverse proxy, but you definitely need hard refresh after changing the picture.

Without any changes on gogs, now all pictures are there, so it was either caching or the proxy. Probably the caching, since I just compared, proxy settings haven't changed.
What is a «hard refresh»? Shouldn't that be done automatically when it's necessary?

@mwaeckerlin https://www.google.com/search?q=What+is+a+«hard+refresh&oq=What+is+a+«hard+refresh&aqs=chrome..69i57&sourceid=chrome&ie=UTF-8

Ah, you don't mean on server side. Sorry for the misunderstanding. At least a force-/shift-reload didn't help. Shouldn't that be fixed in the caching/refresh hints passed by the server in the http-headers?

Keep refreshing until it works. Gogs doesn't cache images, your browser and/or proxy does...

authors: Unknwon justpsvm mwaeckerlin


text:
"git remote show ..." doesn't work

The command to get information on a remote does not seem to work.
When I perform any other command (so far) with Gogs, it works fine as long as my SSH-key is installed correctly.
When I perform this command, it retorts with a message similar to that which is shown when the SSH-key is not installed.
See:
Gogs: given repository does not exist
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Ok, well. Close for now.

authors: Unknwon ghost


text:
Orientation hint fix

There's currently a bug while setting orientation hint in Camera1: current formula doesn't respect the fact that sensor might be installed reversed on the device (like on Nexuses) which causes photo and video rotated in the result file (while preview is actually okay).
Open issues probably related to this bug:
https://github.com/gogopop/CameraKit-Android/issues/14
https://github.com/gogopop/CameraKit-Android/issues/67
Peeped this solution here:
https://github.com/googlesamples/android-Camera2Video/blob/master/Application/src/main/java/com/example/android/camera2video/Camera2VideoFragment.java
Additional minor change is the ability to record video muted. Added style parameter ckAudioEnabled which is true by default.
Disable sound recording feature request:
https://github.com/gogopop/CameraKit-Android/issues/121
P.S. First commit called "Speech recognition audio_type support" because I initially tried to allow SpeechRecognizer work simultaneously with video recording which turned out to be impossible, and I decided just to mute audio.

This is just for video recording?
After compiling this change, Picture result from camera still rotated 90 degree on my Sony Xperia Z5.

@awidiyadew does standard android gallery displays result images rotated as well? This might be exif issue - the bitmap is actually rotated inside result but file contains exif info how to rotate it back.
Usually it's not enough just to read bitmap from file and set it into ImageView. You either need to apply rotation matrix to an ImageView accordingly to exif or to rotate the bitmap you get in result from camera

For me this solution works in all orientations but landscape with facing camera. It needs an additional flip from top to bottom

Hey @yatsinar we really appreciate the work here and look forward to more potential PRs, but due to the merge conflict we won't be merging this in currently. 100% open to reexamining if resolved though.
We have been picking up the pace dramatically lately on CameraKit, look for many more enhancements in the coming months. 0.12.0 is slated to be released this week!
@droidpl @awidiyadew

authors: austinkettner awidiyadew droidpl yatsinar

repo: gogopop/CameraKit-Android

license: mit

license_source: bigquery


text:
add requested state to permissionStatus, fix requested bg location show up

What does this change?
Add requested state to permissionStatus to fix already-requested permissions to show when the service of it is disabled.
How do we test this change?

Make Foreground location and GPS required, Background location optional
Grant foreground location but not background location
Turn off location (GPS)
call show() again

Checklist

[x] Run the pre_pr.sh script to ensure code quality.

temporarily enabled merge commit to fix CodeCov

repo: gogovan/flutter-force-permission


text:
[FEAT] Remove Viper

Viper depends on hcl / hashicorp, which uses the MPL. As a result, we need to bundle the source code into dependencies, and it balloons to 1.5MB (all other deps only require LICENSE/NOTICE). We may desire to integrate HCL support later anyways, but we'd probably do it via a plugin to prevent shipping the HCL repo code in grlx.
Just a thought.

Agreed, HCL might not be what we want to use for configuration. At the start, YAML + JSON (and maybe TOML), would be a good start and would be easy enough to implement and maintain.

We're not currently supporting HCL at all, but it's brought in as a dependency regardless.
If we were to support HCL in the future it would likely be as a plugin/module and more as a deploy integration.
We wouldn't want to allow users to use HCL as a recipe file format.

authors: ethanholz taigrr

repo: gogrlx/grlx


text:
0.56 Out of memory on raspberry

Running Hugo 0.56 runs out of memory on Raspberry Raspbian with 1GB RAM
Linux pi3 4.19.50-v7+ #896 SMP Thu Jun 20 16:11:44 BST 2019 armv7l GNU/Linux
-- added log
Building sites … WARN 2019/07/27 15:15:33 Page's .URL is deprecated and will be removed in a future release. Use .Permalink or .RelPermalink. If what you want is the front matter URL value, use .Params.url.
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0xf71828, 0x16)
/usr/local/go/src/runtime/panic.go:617 +0x5c
runtime.sysMap(0x38c00000, 0x2c00000, 0x1cfa978)
/usr/local/go/src/runtime/mem_linux.go:170 +0xac
runtime.(*mheap).sysAlloc(0x1cead40, 0x2810000, 0x175, 0x35e10)
/usr/local/go/src/runtime/malloc.go:633 +0x180
runtime.(*mheap).grow(0x1cead40, 0x1408, 0x0)
/usr/local/go/src/runtime/mheap.go:1222 +0x2c
runtime.(*mheap).allocSpanLocked(0x1cead40, 0x1408, 0x1cfa988, 0x0)
/usr/local/go/src/runtime/mheap.go:1150 +0x3e8
runtime.(*mheap).alloc_m(0x1cead40, 0x1408, 0x3f2f0101, 0x66bd12a0)
/usr/local/go/src/runtime/mheap.go:977 +0xd0
runtime.(*mheap).alloc.func1()
/usr/local/go/src/runtime/mheap.go:1048 +0x3c
runtime.(*mheap).alloc(0x1cead40, 0x1408, 0x1010101, 0x2)
/usr/local/go/src/runtime/mheap.go:1047 +0x60
runtime.largeAlloc(0x280fc8c, 0x60101, 0x36738000)
/usr/local/go/src/runtime/malloc.go:1055 +0x7c
runtime.mallocgc.func1()
/usr/local/go/src/runtime/malloc.go:950 +0x38
runtime.systemstack(0x28645a0)
/usr/local/go/src/runtime/asm_arm.s:354 +0x84
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1153
goroutine 137 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_arm.s:298 +0x4 fp=0x33cf6bc sp=0x33cf6b8 pc=0x6c488
runtime.mallocgc(0x280fc8c, 0xdab290, 0x1, 0x33cf73c)
/usr/local/go/src/runtime/malloc.go:949 +0x894 fp=0x33cf720 sp=0x33cf6bc pc=0x1b6c4
runtime.makeslice(0xdab290, 0x280fc8c, 0x280fc8c, 0xdab050)
/usr/local/go/src/runtime/slice.go:49 +0x70 fp=0x33cf734 sp=0x33cf720 pc=0x56458
bytes.makeSlice(0x280fc8c, 0x0, 0x0, 0x0)
/usr/local/go/src/bytes/buffer.go:232 +0x58 fp=0x33cf748 sp=0x33cf734 pc=0xed5c0
bytes.(*Buffer).grow(0xdbf3da0, 0x20a15f8, 0xdab050)
/usr/local/go/src/bytes/buffer.go:145 +0x150 fp=0x33cf76c sp=0x33cf748 pc=0xecf28
bytes.(*Buffer).Write(0xdbf3da0, 0x36738000, 0x20a15f8, 0x20a2000, 0x1291401, 0xdab050, 0x3daff8c)
/usr/local/go/src/bytes/buffer.go:175 +0xdc fp=0x33cf780 sp=0x33cf76c pc=0xed224
fmt.Fprint(0x1267b08, 0xdbf3da0, 0x33cf7e4, 0x1, 0x1, 0x3daff01, 0x2d5ee00, 0x0)
/usr/local/go/src/fmt/print.go:234 +0x68 fp=0x33cf7b0 sp=0x33cf780 pc=0xdfbe4
text/template.(*state).printValue(0x33cfa04, 0x1279a68, 0x2d5ef80, 0xdab050, 0x3daff8c, 0x98)
/usr/local/go/src/text/template/exec.go:941 +0xac fp=0x33cf7fc sp=0x33cf7b0 pc=0x18eaa0
text/template.(*state).walk(0x33cfa04, 0xe6c850, 0x3ac8300, 0x194, 0x1279a68, 0x2d5ef80)
/usr/local/go/src/text/template/exec.go:256 +0x430 fp=0x33cf840 sp=0x33cf7fc pc=0x188028
text/template.(*state).walk(0x33cfa04, 0xe6c850, 0x3ac8300, 0x194, 0x1279bd0, 0x2d5e7e0)
/usr/local/go/src/text/template/exec.go:262 +0xec fp=0x33cf884 sp=0x33cf840 pc=0x187ce4
text/template.(*state).walkRange.func1(0xdaa540, 0x1cf9620, 0x82, 0xe6c850, 0x3ac8300, 0x194)
/usr/local/go/src/text/template/exec.go:349 +0x108 fp=0x33cf8a8 sp=0x33cf884 pc=0x197200
text/template.(*state).walkRange(0x3d31a04, 0xf15160, 0x385c060, 0x16, 0x2d66a00)
/usr/local/go/src/text/template/exec.go:358 +0x420 fp=0x33cf93c sp=0x33cf8a8 pc=0x188d28
text/template.(*state).walk(0x3d31a04, 0xf15160, 0x385c060, 0x16, 0x1279c70, 0x2d66a00)
/usr/local/go/src/text/template/exec.go:265 +0x3a0 fp=0x33cf980 sp=0x33cf93c pc=0x187f98
text/template.(*state).walk(0x33cfa04, 0xf15160, 0x385c060, 0x16, 0x1279bd0, 0x2ec34c0)
/usr/local/go/src/text/template/exec.go:262 +0xec fp=0x33cf9c4 sp=0x33cf980 pc=0x187ce4
text/template.(*state).walkTemplate(0x3d31aec, 0xf15160, 0x385c060, 0x16, 0x2d77360)
/usr/local/go/src/text/template/exec.go:412 +0x184 fp=0x33cfa28 sp=0x33cf9c4 pc=0x18919c
text/template.(*state).walk(0x3d31aec, 0xf15160, 0x385c060, 0x16, 0x1279cc0, 0x2d77360)
/usr/local/go/src/text/template/exec.go:267 +0x19c fp=0x33cfa6c sp=0x33cfa28 pc=0x187d94
text/template.(*state).walk(0x33cfaec, 0xf15160, 0x385c060, 0x16, 0x1279bd0, 0x2d76ea0)
/usr/local/go/src/text/template/exec.go:262 +0xec fp=0x33cfab0 sp=0x33cfa6c pc=0x187ce4
text/template.(*Template).execute(0x2e29620, 0x1267b08, 0xdbf3da0, 0xf15160, 0x385c060, 0x0, 0x0)
/usr/local/go/src/text/template/exec.go:217 +0x140 fp=0x33cfb10 sp=0x33cfab0 pc=0x1877c0
text/template.(*Template).Execute(...)
/usr/local/go/src/text/template/exec.go:200
html/template.(*Template).Execute(0x2d77520, 0x1267b08, 0xdbf3da0, 0xf15160, 0x385c060, 0x126ee01, 0x126ee40)
/usr/local/go/src/html/template/template.go:122 +0x6c fp=0x33cfb30 sp=0x33cfb10 pc=0x202b00
github.com/gohugoio/hugo/tpl.(*TemplateAdapter).Execute(0x45d1b00, 0x1267b08, 0xdbf3da0, 0xf15160, 0x385c060, 0x0, 0x0)
/root/project/hugo/tpl/template.go:163 +0x90 fp=0x33cfb78 sp=0x33cfb30 pc=0x730210
github.com/gohugoio/hugo/hugolib.(*Site).renderForLayouts(0x2b9e540, 0xf1c4fe, 0x4, 0xf1b3f6, 0x4, 0xf15160, 0x385c060, 0x1267b08, 0xdbf3da0, 0x3bb2380, ...)
/root/project/hugo/hugolib/site.go:1724 +0x70 fp=0x33cfbc8 sp=0x33cfb78 pc=0x890350
github.com/gohugoio/hugo/hugolib.(*Site).renderAndWritePage(0x2b9e540, 0x2afc3c8, 0x3a8ae8c, 0x15, 0x3ba6a20, 0x2c, 0x385c060, 0x3bb2380, 0x10, 0x10, ...)
/root/project/hugo/hugolib/site.go:1640 +0x110 fp=0x33cfe48 sp=0x33cfbc8 pc=0x88fde4
github.com/gohugoio/hugo/hugolib.pageRenderer(0x3c6fe20, 0x2b9e540, 0x3234c40, 0x37a36c0, 0x3cea010)
/root/project/hugo/hugolib/site_render.go:152 +0x428 fp=0x33cffd4 sp=0x33cfe48 pc=0x892708
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:868 +0x4 fp=0x33cffd4 sp=0x33cffd4 pc=0x6e1b0
created by github.com/gohugoio/hugo/hugolib.(*Site).renderPages
/root/project/hugo/hugolib/site_render.go:73 +0x104
goroutine 1 [chan send]:
github.com/gohugoio/hugo/hugolib.(*Site).renderPages(0x2b9e540, 0x3c6fe20, 0x0, 0x0)
/root/project/hugo/hugolib/site_render.go:90 +0x1ec
github.com/gohugoio/hugo/hugolib.(*Site).render(0x2b9e540, 0x3c6fe20, 0x0, 0x0)
/root/project/hugo/hugolib/site.go:1163 +0x44
github.com/gohugoio/hugo/hugolib.(*HugoSites).render(0x2bce480, 0x30cd8e0, 0x33cdc68, 0x18)
/root/project/hugo/hugolib/hugo_sites_build.go:308 +0x2fc
github.com/gohugoio/hugo/hugolib.(*HugoSites).Build.func4()
/root/project/hugo/hugolib/hugo_sites_build.go:127 +0x2c
runtime/trace.WithRegion(0x1276820, 0x30cdf40, 0xf23f7d, 0x6, 0x33cdd04)
/usr/local/go/src/runtime/trace/annotation.go:137 +0xe0
github.com/gohugoio/hugo/hugolib.(*HugoSites).Build(0x2bce480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/root/project/hugo/hugolib/hugo_sites_build.go:129 +0x588
github.com/gohugoio/hugo/commands.(*commandeer).buildSites(...)
/root/project/hugo/commands/hugo.go:707
github.com/gohugoio/hugo/commands.(*commandeer).fullBuild.func3(0x0, 0x0)
/root/project/hugo/commands/hugo.go:318 +0x4c
github.com/gohugoio/hugo/commands.(*commandeer).fullBuild(0x2ab7b90, 0x0, 0x0)
/root/project/hugo/commands/hugo.go:330 +0x12c
github.com/gohugoio/hugo/commands.(*commandeer).build(0x2ab7b90, 0x0, 0x0)
/root/project/hugo/commands/hugo.go:479 +0xc0
github.com/gohugoio/hugo/commands.(*commandsBuilder).newHugoCmd.func1(0x2a78c80, 0x2a53580, 0x0, 0x7, 0x0, 0x0)
/root/project/hugo/commands/commands.go:162 +0xb0
github.com/spf13/cobra.(*Command).execute(0x2a78c80, 0x2824148, 0x7, 0x7, 0x2a78c80, 0x2824148)
/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20190321000552-67fc4837d267/command.go:762 +0x360
github.com/spf13/cobra.(*Command).ExecuteC(0x2a78c80, 0x2ab2138, 0x2, 0x1)
/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20190321000552-67fc4837d267/command.go:850 +0x234
github.com/gohugoio/hugo/commands.Execute(0x2824148, 0x7, 0x7, 0x40770, 0x28320b0, 0x0, 0x0)
/root/project/hugo/commands/hugo.go:89 +0x8c
main.main()
/root/project/hugo/main.go:23 +0x58
goroutine 7 [select]:
go.opencensus.io/stats/view.(*worker).start(0x297a180)
/go/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:154 +0xb0
created by go.opencensus.io/stats/view.init.0
/go/pkg/mod/go.opencensus.io@v0.22.0/stats/view/worker.go:32 +0x48
goroutine 8 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0x130
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x14
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x30
goroutine 136 [runnable]:
bytes.(*Buffer).WriteString(0x3378ea0, 0xb0f2000, 0x87a9c, 0x76f82a30, 0x0, 0x3744298)
/usr/local/go/src/bytes/buffer.go:183 +0x110
html/template.processURLOnto(0xb0f2000, 0x87a9c, 0x1, 0x3378ea0, 0x87a9c)
/usr/local/go/src/html/template/url.go:140 +0x388
html/template.urlProcessor(0x3a2f701, 0x37a7b30, 0x1, 0x1, 0x3378e8c, 0x23114)
/usr/local/go/src/html/template/url.go:82 +0x8c
html/template.urlNormalizer(0x37a7b30, 0x1, 0x1, 0x0, 0x0)
/usr/local/go/src/html/template/url.go:71 +0x34
reflect.Value.call(0xd85db0, 0xffc818, 0x13, 0xf1b276, 0x4, 0x3c1cdc0, 0x1, 0x1, 0x31ac1c0, 0x2842b40, ...)
/usr/local/go/src/reflect/value.go:447 +0x3a4
reflect.Value.Call(0xd85db0, 0xffc818, 0x13, 0x3c1cdc0, 0x1, 0x1, 0x18b9c0, 0x10, 0x18bf94)
/usr/local/go/src/reflect/value.go:308 +0x74
text/template.safeCall(0xd85db0, 0xffc818, 0x13, 0x3c1cdc0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/text/template/funcs.go:293 +0x90
text/template.(*state).evalCall(0x45e1a04, 0xe6c850, 0x2ded438, 0x194, 0xd85db0, 0xffc818, 0x13, 0x1279b08, 0x3f39be0, 0xf7eeef, ...)
/usr/local/go/src/text/template/exec.go:703 +0x508
text/template.(*state).evalFunction(0x45e1a04, 0xe6c850, 0x2ded438, 0x194, 0x3f39bc0, 0x1279b08, 0x3f39be0, 0x39424c0, 0x1, 0x1, ...)
/usr/local/go/src/text/template/exec.go:564 +0xf4
text/template.(*state).evalCommand(0x45e1a04, 0xe6c850, 0x2ded438, 0x194, 0x3f39be0, 0xdab050, 0x3378e48, 0x98, 0xdab050, 0x3378e48, ...)
/usr/local/go/src/text/template/exec.go:461 +0x560
text/template.(*state).evalPipeline(0x45e1a04, 0xe6c850, 0x2ded438, 0x194, 0x2ec61e0, 0x0, 0x0, 0x2d5ee40)
/usr/local/go/src/text/template/exec.go:430 +0xd8
text/template.(*state).walk(0x45e1a04, 0xe6c850, 0x2ded438, 0x194, 0x1279a68, 0x2d5ef80)
/usr/local/go/src/text/template/exec.go:254 +0x3ec
text/template.(*state).walk(0x45e1a04, 0xe6c850, 0x2ded438, 0x194, 0x1279bd0, 0x2d5e7e0)
/usr/local/go/src/text/template/exec.go:262 +0xec
text/template.(*state).walkRange.func1(0xdaa540, 0x495408c, 0x82, 0xe6c850, 0x2ded438, 0x194)
/usr/local/go/src/text/template/exec.go:349 +0x108
text/template.(*state).walkRange(0xe60fa04, 0xf15160, 0x30eda40, 0x16, 0x2d66a00)
/usr/local/go/src/text/template/exec.go:358 +0x420
text/template.(*state).walk(0xe60fa04, 0xf15160, 0x30eda40, 0x16, 0x1279c70, 0x2d66a00)
/usr/local/go/src/text/template/exec.go:265 +0x3a0
text/template.(*state).walk(0x45e1a04, 0xf15160, 0x30eda40, 0x16, 0x1279bd0, 0x2ec34c0)
/usr/local/go/src/text/template/exec.go:262 +0xec
text/template.(*state).walkTemplate(0xe60faec, 0xf15160, 0x30eda40, 0x16, 0x2d77360)
/usr/local/go/src/text/template/exec.go:412 +0x184
text/template.(*state).walk(0xe60faec, 0xf15160, 0x30eda40, 0x16, 0x1279cc0, 0x2d77360)
/usr/local/go/src/text/template/exec.go:267 +0x19c
text/template.(*state).walk(0x45e1aec, 0xf15160, 0x30eda40, 0x16, 0x1279bd0, 0x2d76ea0)
/usr/local/go/src/text/template/exec.go:262 +0xec
text/template.(*Template).execute(0x2e29620, 0x1267b08, 0x6bd56c0, 0xf15160, 0x30eda40, 0x0, 0x0)
/usr/local/go/src/text/template/exec.go:217 +0x140
text/template.(*Template).Execute(...)
/usr/local/go/src/text/template/exec.go:200
html/template.(*Template).Execute(0x2d77520, 0x1267b08, 0x6bd56c0, 0xf15160, 0x30eda40, 0x126ee01, 0x126ee40)
/usr/local/go/src/html/template/template.go:122 +0x6c
github.com/gohugoio/hugo/tpl.(*TemplateAdapter).Execute(0x3c6cff0, 0x1267b08, 0x6bd56c0, 0xf15160, 0x30eda40, 0x0, 0x0)
/root/project/hugo/tpl/template.go:163 +0x90
github.com/gohugoio/hugo/hugolib.(*Site).renderForLayouts(0x2b9e540, 0xf1c4fe, 0x4, 0xf1b3f6, 0x4, 0xf15160, 0x30eda40, 0x1267b08, 0x6bd56c0, 0x3bb2380, ...)
/root/project/hugo/hugolib/site.go:1724 +0x70
github.com/gohugoio/hugo/hugolib.(*Site).renderAndWritePage(0x2b9e540, 0x2afc3c8, 0xe60fe8c, 0x14, 0x326ced0, 0x2c, 0x30eda40, 0x3bb2380, 0x10, 0x10, ...)
/root/project/hugo/hugolib/site.go:1640 +0x110
github.com/gohugoio/hugo/hugolib.pageRenderer(0x3c6fe20, 0x2b9e540, 0x3234c40, 0x37a36c0, 0x3cea010)
/root/project/hugo/hugolib/site_render.go:152 +0x428
created by github.com/gohugoio/hugo/hugolib.(*Site).renderPages
/root/project/hugo/hugolib/site_render.go:73 +0x104
goroutine 127 [chan receive]:
github.com/gohugoio/hugo/hugolib.(*HugoSites).Build.func1(0x2bce480, 0x3461600, 0x2d9c200)
/root/project/hugo/hugolib/hugo_sites_build.go:49 +0x68
created by github.com/gohugoio/hugo/hugolib.(*HugoSites).Build
/root/project/hugo/hugolib/hugo_sites_build.go:46 +0x150
goroutine 135 [runnable]:
github.com/tdewolff/parse/v2.QuoteEntity(0x6a4e252, 0xa478f, 0x116abe, 0x3654000, 0x0)
/go/pkg/mod/github.com/tdewolff/parse/v2@v2.3.5/common.go:193 +0x320
github.com/tdewolff/parse/v2/html.EscapeAttrVal(0x284f908, 0x2a918ed6, 0x1049e4, 0x87a72a, 0x69ee000, 0x1049e1, 0x176d10, 0x3654000, 0x2b90ba0, 0x0)
/go/pkg/mod/github.com/tdewolff/parse/v2@v2.3.5/html/util.go:64 +0xe0
github.com/tdewolff/minify/v2/html.(*Minifier).Minify(0x2a8e030, 0x2be3d00, 0x1267b08, 0x6588e20, 0x1267af0, 0x473e240, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/tdewolff/minify/v2@v2.3.7/html/html.go:445 +0x11d8
github.com/tdewolff/minify/v2.MinifierFunc.Minify(...)
/go/pkg/mod/github.com/tdewolff/minify/v2@v2.3.7/minify.go:29
github.com/gohugoio/hugo/minifiers.Client.Transformer.func1(0x126ee88, 0x2b8e110, 0x6588e20, 0xbe70ea)
/root/project/hugo/minifiers/minifiers.go:53 +0x9c
github.com/gohugoio/hugo/transform.(*Chain).Apply(0x284fb10, 0x1267b08, 0x37730e0, 0x1267af0, 0x4e18060, 0x0, 0x0)
/root/project/hugo/transform/chain.go:105 +0x1cc
github.com/gohugoio/hugo/publisher.DestinationPublisher.Publish(0x1281348, 0x2b3dd80, 0x35f9501, 0x2be3d00, 0x1267af0, 0x4e18060, 0xf1b3f6, 0x4, 0xf1c8d6, 0x4, ...)
/root/project/hugo/publisher/publisher.go:100 +0x2a8
github.com/gohugoio/hugo/hugolib.(*Site).renderAndWritePage(0x2b9e540, 0x2afc3c8, 0x3d35e8c, 0x1b, 0x3c66100, 0x33, 0x36f57c0, 0x3bb2380, 0x10, 0x10, ...)
/root/project/hugo/hugolib/site.go:1689 +0x294
github.com/gohugoio/hugo/hugolib.pageRenderer(0x3c6fe20, 0x2b9e540, 0x3234c40, 0x37a36c0, 0x3cea010)
/root/project/hugo/hugolib/site_render.go:152 +0x428
created by github.com/gohugoio/hugo/hugolib.(*Site).renderPages
/root/project/hugo/hugolib/site_render.go:73 +0x104
goroutine 134 [runnable]:
github.com/tdewolff/parse/v2.QuoteEntity(0xe228363, 0xacd4e, 0x1b1b61, 0x13ce0000, 0x0)
/go/pkg/mod/github.com/tdewolff/parse/v2@v2.3.5/common.go:193 +0x320
github.com/tdewolff/parse/v2/html.EscapeAttrVal(0xe60b908, 0x31f82891, 0xef0b5, 0xf7b56f, 0xe1e6000, 0xef0b1, 0x1f3ec4, 0x3ca0f40, 0x2b90ba0, 0x0)
/go/pkg/mod/github.com/tdewolff/parse/v2@v2.3.5/html/util.go:110 +0x39c
github.com/tdewolff/minify/v2/html.(*Minifier).Minify(0x2a8e030, 0x2be3d00, 0x1267b08, 0x4146000, 0x1267af0, 0x41470c0, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/tdewolff/minify/v2@v2.3.7/html/html.go:445 +0x11d8
github.com/tdewolff/minify/v2.MinifierFunc.Minify(...)
/go/pkg/mod/github.com/tdewolff/minify/v2@v2.3.7/minify.go:29
github.com/gohugoio/hugo/minifiers.Client.Transformer.func1(0x126ee88, 0x3724000, 0x4146000, 0xd31082)
/root/project/hugo/minifiers/minifiers.go:53 +0x9c
github.com/gohugoio/hugo/transform.(*Chain).Apply(0xe60bb10, 0x1267b08, 0x4146900, 0x1267af0, 0x3553e00, 0x0, 0x0)
/root/project/hugo/transform/chain.go:105 +0x1cc
github.com/gohugoio/hugo/publisher.DestinationPublisher.Publish(0x1281348, 0x2b3dd80, 0x3f04601, 0x2be3d00, 0x1267af0, 0x3553e00, 0xf1b3f6, 0x4, 0xf1c8d6, 0x4, ...)
/root/project/hugo/publisher/publisher.go:100 +0x2a8
github.com/gohugoio/hugo/hugolib.(*Site).renderAndWritePage(0x2b9e540, 0x2afc3c8, 0xe60be8c, 0x10, 0x369cf30, 0x28, 0x3798040, 0x8bc8f00, 0x8, 0x8, ...)
/root/project/hugo/hugolib/site.go:1689 +0x294
github.com/gohugoio/hugo/hugolib.pageRenderer(0x3c6fe20, 0x2b9e540, 0x3234c40, 0x37a36c0, 0x3cea010)
/root/project/hugo/hugolib/site_render.go:152 +0x428
created by github.com/gohugoio/hugo/hugolib.(*Site).renderPages
/root/project/hugo/hugolib/site_render.go:73 +0x104
goroutine 133 [chan receive]:
github.com/gohugoio/hugo/hugolib.(*Site).errorCollator(0x2b9e540, 0x37a36c0, 0x37a3700)
/root/project/hugo/hugolib/site.go:1566 +0x5c
created by github.com/gohugoio/hugo/hugolib.(*Site).renderPages
/root/project/hugo/hugolib/site_render.go:67 +0x98

not gone with 0.56.1
If I delete ./recources the error comes back for some runs.
--> could this be a memory leak in image processing?
I only use .Fit   and  .Content | base64Encode

repo: gohugoio/hugo


text:
Feast API: Sources

This issue can be used to discuss the role of sources in Feast, and how we see the concept evolving in future versions.
Status quo
Feast currently supports only a single Source type on KafkaSource. This can be defined through a Feature Set or omitted. If users omit the source, Feast Core will fill in a default for the user.
Contributor comment from @ches (link)

While I get the utility and convenience especially for deploying Feast with more distributed ownership in the organization, we ignore / work around a lot of the Source and Store configuration kept in the registry RDBMS, in favor of service discovery.

I also think that asking data scientists and data engineers to be concerned with operational infra configuration like Kafka broker addresses when registering feature sets is not an elegant separation of concerns.

Maybe we can take this as an opportunity to consider design alternatives as well.

we ignore / work around a lot of the Source and Store configuration kept in the registry RDBMS, in favor of service discovery.

By "we" do you mean your current team? I'd love to get a better understanding of how you are currently doing it, especially if it means we can improve sources or stores.

I also think that asking data scientists and data engineers to be concerned with operational infra configuration like Kafka broker addresses when registering feature sets is not an elegant separation of concerns.

I agree 100%, but it depends on what aspect of sources you are referring to.
Part of the reason why we made sources optional was exactly for this reason. We didn't want a data scientist to have to think about configuring a source. In hindsight this created a lot of complexity for us that I think we should have delayed, but here we are.
Sources today are a glorified "connection string". If a source was to only ever stay a connection string, then I don't really see the point in exposing that to users. This can easily be configured behind the scenes by administrators and exposed through source names or a more human friendly way. Users can then select the source they want to use. If I understand correctly then this is the largest part of what you think is a bad separation of concern.
However, I do think there is value in having users configure some aspects of the sourcing of data. Perhaps source isn't the right term, but we can discuss that later.
Most of our data still lives in data lakes or data warehouses, which means federation is a natural next step. In a federated model we would probably opt to extend sources to allow new data sources to be accessed through Feast, especially without users having to export and reimport into Feast, and with Feast being lazy towards retrieval and exports (no long running jobs).
The next question is then, how does this influence the Feast API?
One extreme is the Uber approach towards data transformation. There are parts of that API that I like, and parts that I don't like. I think we are in agreement that we shouldn't be building a general purpose data transformation system, for example.
However, I see massive value in allowing users to define SQL queries. And this is the direction that I would like to take sources (or if not sources, another part of feature sets).
One of the main reasons why I see this as valuable is that all of our users are familiar with SQL. Using SQL improves the Feast user experience because they are able to validate and prove that their query works without Feast in the loop. They can then bring that query to Feast, publish it, and see the results. If there is a failure, then the problem is likely Feast. SQL is also supported by virtually all sources and stores.
name: driver_features
entities:
- name: driver_id
  type: INT64
features:
- name: avg_speed
  valueType: FLOAT
- name: avg_daily_trips
  valueType: FLOAT
sources:
  batch:
    type: BigQuerySQL
    sql: |
    SELECT driver_id, avg_speed, avg_daily_trips from `gcp_project.dataset.table`

There is a bit of leakiness here in that the project, dataset, and table are leaked through. In theory we could abstract away all of this SQL, but it would require a bit more development effort and detract from the development experience.

Thanks for the thoughtful reply @woop. I didn't pay close attention to the 0.5 milestone on #632 so thanks for diverting it here, makes sense to open this discussion issue that parallels other API ones.

By "we" do you mean your current team? I'd love to get a better understanding of how you are currently doing it, especially if it means we can improve sources or stores.

Yes, at Agoda our Feast deployment is perhaps more centralized in the sense that:

There is one Store for warehouse (HDFS with Hive-interoperable engines)
There is one Store for online serving (Cassandra)
There is logically one Source Kafka topic for all feature ingestion (modulo internal idiosyncrasies not pertinent to the discussion)
Feast is offered to all clients as a managed service provided by one team, client teams do not deploy their own Stores or Serving API instances.

As I'm sure is true for Gojek too, some core entity types in our business domain have very high cardinality (e.g. customers). Most client teams serving online will use some features of these, and it isn't practical or economical for us to deploy many storage cluster islands that can support the scale. Cassandra is massively scalable; Kafka throughput is massively scalable; we have dedicated teams expert at doing those. There's also the merit that new clients don't need to provision new infrastructure to start using the system, this is one of the key problems we're solving from the status quo before Feast. (We track cost attribution in other ways, if anyone wonders about that aspect).
The one point above that I imagine could become more flexible over time is the Kafka topics, there may be use cases for special-purpose / priority ones, and I believe it should be straightforward to support that if the need arises.
That brings up a notable distinction in regard to the current Source configuration I think: if we did support this, it will be useful to declaratively associate feature sets with source topics (as Feast already allows). However, users will never need to think about the brokers, they will differ for the same topic name across DCs and our SDK wrapper gets them from service discovery. I think this speaks to your thought that "there is value in having users configure some aspects of the sourcing of data".

Sources today are a glorified "connection string". If a source was to only ever stay a connection string, then I don't really see the point in exposing that to users. This can easily be configured behind the scenes by administrators and exposed through source names or a more human friendly way. Users can then select the source they want to use. If I understand correctly then this is the largest part of what you think is a bad separation of concern.

Yes I believe we're on the same page then. Roughly, an abstraction over operational or environmental details of infrastructure. Operators of a Feast deployment could plug service discovery into this abstraction, potentially.

Most of our data still lives in data lakes or data warehouses, which means federation is a natural next step. In a federated model we would probably opt to extend sources to allow new data sources to be accessed through Feast, especially without users having to export and reimport into Feast, and with Feast being lazy towards retrieval and exports (no long running jobs).

I feel the federation is an elegant idea in theory, but I'm initially skeptical of how it will work out in practice. Not to say it isn't worth trying or to discourage it, just would urge breaking it off to an MVP to validate without disrupting Feast's ongoing technical improvement and data model refinement—it could be a year spent rearchitecting for such a pivot in vision, with considerable risk that it doesn't work out well or serve users markedly better.
Some of my concerns with it:

"Jack of all trades, master of none" being difficult to give consistently optimal experience with multiple storage engines, e.g. not being able to push down filters and joins across stores. See also areas like #444 and more specifically the data locality-related discussion on #482 that I think possibly belongs under #444.
Harder (and less efficient) to lazily impose data quality measures on data at rest.
Hard to rely on sustained availability of data from sources controlled by many owners.

We may learn differently with more experience, but at the outset in our org I think we are content to bring data into managed feature store storage. There's a cultural expectation that it is "special" data, expected to be subjected to higher quality standards, stable maintenance, etc. that federated sources may not.

However, I see massive value in allowing users to define SQL queries. And this is the direction that I would like to take sources (or if not sources, another part of feature sets).
One of the main reasons why I see this as valuable is that all of our users are familiar with SQL. Using SQL improves the Feast user experience because they are able to validate and prove that their query works without Feast in the loop. They can then bring that query to Feast, publish it, and see the results. If there is a failure, then the problem is likely Feast. SQL is also supported by virtually all sources and stores.

I'm on board with looking for ways to use SQL as an interface to the system. It does make barriers vanish for many potential users, especially data scientists/engineers/analysts who can contribute new data sets to the feature store without more specialized development knowledge/skills. Indeed something that has happened even before Feast going live for us was another team eagerly building an integration with an in-house ETL tool we have to move data between engines with—you guessed it—SQL expressions of the input. So we've already "solved" this, in a proprietary way and with some overhead of redundant import/exports that you refer to with federation.
We (at least I 😇) have a vision/dream of a streaming platform where users are expressing Beam/Flink/Spark/Whatever SQL, with ability to include/join feature store data, and (optionally) choosing to ingest results into the feature store in the same engine DAG (no extra hop out through Kafka or the like). In theory we are not that far from the query part, the data is already in tables the engine can make available in PCollections or their analogues.
I may have lost the course a little bit there, but hopefully it gives color to ubiquity of SQL.

at Agoda our Feast deployment is perhaps more centralized

This really strikes a chord on another note for me, perhaps outside of the Source discussion. I hope you will entertain this rant.
The notion of centralization versus decentralization. The question that I have in my mind is whether we ever even need multiple historical stores per Feast Core, or just one. I can see the need for multiple serving stores, but at least in the case of Gojek we can probably get away with just one.
Why is this important?
Well what happens when we move job management to Feast Serving? We will have a smart serving layer with its own persistence, and presumably Core will become nothing more than a registry. This makes a lot of sense in terms of local execution, access management, connectivity, safe upgrades, cost attribution.
However, it falls apart in another way. If a user ingests data, where do they ingest to? Do they ingest straight to one Serving deployment or to topics managed by Core? If Serving, then it means it's a localized ingestion that other stores don't benefit from. If Core, then it means Core still does data management (or stream management).
Also, where would a user interface go? If it sits on Core then which historical store's data does it show? Or does it show data from all stores? Users would find it very counter-intuitive if the feature store wasn't a single source of truth. The same holds true for statistics. We won't pull statistics from online stores, and we only run a single historical store, yet we are asking users to provide the store name because we theoretically support multiple historical stores.
Anyway, the bottom line is I think a lot of problems could be solved by only having a single historical store per Feast Core. If you keep poking holes at this you might wonder why Feast Serving (batch/historical) and Feast Core are even separate services in the first place.

and it isn't practical or economical for us to deploy many storage cluster islands that can support the scale.

I wish this was the case for us. We currently have to manage our own storage clusters unfortunately, which is becoming a little bit of a pain. We'll probably be doubling down on Redis Cluster once 0.5 goes live, and try to keep things as simple as possible.

However, users will never need to think about the brokers, they will differ for the same topic name across DCs and our SDK wrapper and Ingestion get them from service discovery. I think this speaks to your thought that "there is value in having users configure some aspects of the sourcing of data".

In some ways this ties to what I was saying above about having one historical store and what the architecture looks like when/if Core is made into a pure registry. Let's assume you have centrally registered feature sets, presumably you want to set configuration on the Serving deployment that can somehow uniquely configure the brokers, because Core isn't in the remote DC. However, that means ingestion doesn't make sense as a responsibility of Feast Core, it must then fall to Serving, unless you will have multiple Feast Core deployments. It feels a lot cleaner to me to have Feast Serving manage ingestion (and interpolate brokers based on local config) than have that centrally managed.

I feel the federation is an elegant idea in theory, but I'm initially skeptical of how it will work out in practice. Not to say it isn't worth trying or to discourage it, just would urge breaking it off to an MVP to validate without disrupting Feast's ongoing technical improvement and data model refinement—it could be a year spent rearchitecting for such a pivot in vision, with considerable risk that it doesn't work out well or serve users markedly better.

Given our track record of hitting deadlines, I am just as weary about scope creep here. Just to be clear, I would not pick up federation as the sole way of dealing with data, and I would not have it started if we can't build an MVP off the critical path.

"Jack of all trades, master of none" being difficult to give consistently optimal experience with multiple storage engines, e.g. not being able to push down filters and joins across stores. See also areas like #444 and more specifically the data locality-related discussion on #482 that I think possibly belongs under #444.

This is a valid concern. I'd want to limit the scope of this implementation as much as possible, and potentially add constraints. Let's take the BQ example above. MVP example: For every user that created a feature set with an external BQ source we create (1) a BigQuery view to the external table and (2) jobs that occasionally update online stores from this view.
What does that give us? Well it saves us a lot of money on Dataflow jobs and means we can manage less infrastructure. It means users don't have to ingest data (and can simply plug in their queries). On both online and batch serving nothing changes.

Harder (and less efficient) to lazily impose data quality measures on data at rest.
This is true, but there is also a benefit that externalizing data ownership means the feature store doesn't become the source of truth. It reduces the operational burden by allowing the feature store team to be able to "rebuild" their stores without much risk.

Hard to rely on sustained availability of data from sources controlled by many owners—producers controlling fate of consumers without a safety buffer.

I agree with this, but this problem doesn't go away in Gojek's case at least. Essentially we are adding one more step to "federation" by having the user extract the data and publish it. However, if the user is savvy then they will just propagate the problems from the source straight to Feast. Then, when something goes wrong they will ask why Feast doesn't have the right data. Tracing this back upstream then becomes harder because Feast doesn't have any context outside of rows landing on a stream.
I'm not saying one approach is definitely superior, but I do think that there are trade-offs here.

risk that it doesn't work out well or serve users markedly better

I think federation in and of itself doesn't really add much value to end-users. It adds value to the operators. However Feast implements "external data sourcing" can change over time, as long as the development API that we expose is more useful to our users, then they will be happy. I can see that being the case with us exposing SQL through sources, especially on the batch only side where our users are less savvy.

There's a cultural expectation that it is "special" data, expected to be subjected to higher quality standards, stable maintenance, etc. that federated sources may not.

I like this. I think we have touched on this idea before. I am hoping that the tooling that we develop on statistics and validation will eventually lend itself to higher trust from our users here, but many of our data scientists have asked for more tools around ingestion. They don't want to push bad data into Feast, so they want to know whether it is good or bad prior to pushing. Or they want tools to undo a bad ingestion once it fails or if the statistics look off. Something to ponder about.

We (at least I innocent) have a vision/dream of a streaming platform where users are expressing Beam/Flink/Spark/Whatever SQL, with ability to include/join feature store data, and (optionally) choosing to ingest results into the feature store in the same engine DAG (no extra hop out through Kafka or the like). In theory we are not that far from the query part, the data is already in tables the engine can make available in PCollections or their analogues.

This sounds somewhat similar to what we have internally right now, but the integration with Feast does require that extra hop through Kafka.
The value add I can see right now is not so much on the streaming side, however. Even though that aspect is a dream state to aim for. We have enough tools to allow users to create new streams and streaming transformations and get them into Feast.

The value add I can see right now is not so much on the streaming side, however. Even though that aspect is a dream state to aim for. We have enough tools to allow users to create new streams and streaming transformations and get them into Feast. I think the part that is still a bit painful is the export/import flow, but perhaps that can be made easier through better controls at ingestion time.

Oh yeah, I should have made clearer that in that dream vision I don't see Feast being the whole thing, but a component of it.

I have been thinking about some recurring issues related to sources, job management, stores, and streams. These issues are not singular but they have bled through the questions people ask in various proposals.
Please see this document for architecture diagrams that illustrate the disinctions that I am making.
Problems

Disconnect between pre- and post-stream ingestion jobs (producer, consumer/populator). Users push into a void (stream), but there is no continuity between the data a user pushes and the jobs that consume/populate stores. If failure happens our users dont see feedback, and its even hard for us to provide that. They have no way to debug ingestion.
Adding more online serving instances requires an increasing amount of complex population/consumption jobs to be managed by Feast Core. These jobs only fulfill ½ of the ingestion process, and all of the complexity is after (not prior) to the streams. This means we have to use Beam between the feature streams and stores that they write to, as opposed to simple consumers.
The idea of a Source is confusing as its implemented. As implemented it is the source of a job that writes into a store, but the original idea  was that the source would be external to Feast and the destination sink would be the Feature Stream. However as implemented, the Source is sometimes the Feature Stream and sometimes an external stream, which means we dont differentiate the Feast Kafka stream from other streams. This means that implicitly there is an explosion of complex population jobs if there is an explosion of consumers, but much of this logic only needs to be applied once on a stream, not for every consumers (Feast Serving).
Moving large amounts of data through streams is wasteful if it isnt used for online serving.

Proposal

Impose a limit of only a single historical store per Feast deployment. This solves multiple problems like where to generate batch statistics, where to point a user interface at (as opposed to selecting one of many). But this constraint is imposed for ingestion purposes in this context.
Ingestion is the act of taking data from a source and writing it into a sink. Sources and sinks are now defined as:

Source types

Anything (bounded, unbounded).

Sink types

The historical warehouse store (BQ, Hive). There will only be 1.
Feature stream

An ingestion job writes to both the stream and historical store. This job is a Beam/Spark job that is triggered through the Feast API (eventually). It could also be two jobs, but that should be abstracted from the user.
An ingestion job is successful when it has written to both sinks. User gets feedback immediately. This solves the problem of user feedback and duality of ingestion jobs.
Ingestion jobs contain all complexity (statistics generation, stream transforms, validation).
Data gets into online stores through the Feature Stream. Data is pulled into the online serving databases (Redis for example) through consumers jobs.

These consumer jobs are deployed alongside each Feast Serving deployment as side cars or they run in process with Feast Serving.
Does not run Beam and does not require complex ingestion jobs on Dataflow. They run local consumers.
They stream in data only from the central Feature Stream. Can be deployed by any team, even end users. Any person that can stand up a docker container can hook into the stream and populate their online serving instance.
Topics are determined for consumption by querying Feast Core.
Feast has no awareness of serving deployments and they don't affect ingestion.
Users can still connect to serving deployments through the SDK in order to see the state of ingestion, especially as it relates to specific ingestion ids.

Please see this document for more details on this proposal.

authors: ches woop

repo: gojek/feast


text:
Error in handling Accept-Encoding

We found an issue with how turing uses Accept-Encoding. We saw turing adding  Accept-Encoding: gzip to outbound requests, but not parsing the response as gzip. This results in clients calling turing receiving gzipped response even when they did not request for it.
Reproduction ->

deploy a service that supports gzip response encoding
configure turing to route to that end point
make a request through Turing without setting Accept-Encoding

cc @peterjrichens

Hi @ankurs, thanks for reporting this! After a quick scrolling through the codebase, I can't see that Turing explicitly adds any extra headers besides what is passed by the client. But the team will look into it and get back with more details soon.

I have a feeling this might be because of https://github.com/gojek/fiber but haven't looking deeper into it
Please let me know if need any more info for this

authors: ankurs romanwozniak

repo: gojek/turing