text:
Failed to build for SwiftUI preview

Hello,
when I add the SwiftAA lib to my SwiftUI app, the live canvas preview fails to build:
symbol(s) not found for architecture x86_64
Here is the full error log:
build-error-swift-UI.txt
From what I understand, it looks like the ObjcAA framework is not built for x86_64 which the SwiftUI  preview  uses.
Did anyone run into the same problem ?
Since I am new to iOS development and struggled to just get the lib to work in a normal build, it is quite possible that I misconfigured something, so any advice would be appreciated :)
Thanks
Edit: forgot to mention the lib was added in Xcode via the Swift Package Manager (full disclosure: tried with Carthage but got another kind of error at runtime)

Hi. I struggled quite a lot to make the project work with all packaging managers. But I haven't tried to setup a SwiftUI setup (simply because I had no real time to try it). But I'll try!

I've made some progresses. There were 2 problems:

it seems that standard Objective-C exception cannot be used inside ObjC++ modules, and I've thus replaced them with standard C++ ones.
There was a forbidden "+" character in the AA+ bundle name. I changed the "AA+" package lib to "AAplus".

Could you give a try and let me know you have a new error ? (something related to NSObject)?

Choose "master" branch in your SwiftAA package dependency in your project (instead of a fixed version range)
Make sure to refresh to the latest version
Remove the "AA+" lib dependency in the "Framrworks, Libraries and Embedded Content" section of your project settings (it doesn't exist anymore)
Manually add the new "AAplus" in place.

Hm, there is something I don't get. When adding only SwiftAA as dependency of my SwiftUI project, everything compiles fine and it runs on simulator. The problem occurs only when one includes ObjCAA. Please, could you try by specifying only SwiftAA and see how it goes ? (it seems to work both with master branch and existing 2.2.2 release).

Hi,
thanks for following up on this.
When I used Swift Package Manager to add SwiftAA, I only selected "SwiftAA", so it is already the only one I have in my Framework list (the project does compile and run fine like this, my only problem is with the SwiftUI preview).
I tried switching to the master branch and tried with and without "AAplus". I got the same error in the preview in both cases. The error log did change a little compared to what I had previously so I'm sending the new one: build-error-swift-UI-2.txt
Just to make sure we are trying the same thing,  the preview I have a problem with is activated like this:

And the error looks like this:

I think it is a fairly recent addition to XCode and it requires macOS Catalina to run, so maybe we do not get the same result because we are on different versions of XCode ? I am using the latest I think (Version 11.3.1 (11C504))

Wonderful, with screenshots. Will check as soon as I can (very busy during the week). I'm also on Catalina.

I’m experiencing the same issue. For some reason Xcode is unable to link against a dynamic library with objc runtime when previewing SwiftUI views. Makes working with SwiftUI difficult because one of the premises of SwiftUI is you write code which you can instantly preview without running your app.
Setting the ObjCAA product’s type to .static solves the issue. As SwiftAA is being statically linked with both ObjCAA and AAplus for whatever reason even when both libraries are set dynamic, then this setting doesn’t change the resulting product.

Really sorry for having let the time flew on this issue... Just overwhelmed with day job. Would you mind making a PR ? I'd be happy to merge rapidly.

Sure, no problem. I’m still baffled it’s not building all libraries as dynamic as it should. I would prefer no to link SwiftAA dependencies statically, so I’ll keep looking for a better solution to solve the issue. If I don’t find one, I’ll make a PR.

v2.2.5 available. @sbenitezb @Saturnyn If you build something with SwiftAA, I'd be happy to include it in the list of apps using it in the README.

authors: Saturnyn onekiloparsec sbenitezb

repo: onekiloparsec/SwiftAA


text:
Different names in Package.swift and Podfile

The inconsistency in naming (Rainbow / RainbowSwift) makes it difficult to build the same project using Xcode and SPM, without editing imports to one or the other.

Why do you need to build it with both SPM and CocoaPods?

I'm generating code coverage data with Xcode but building and running with SPM.

Since the name of Rainbow is already taken by another framework in the central CocoaPods repo, so we could not use that name anymore.
A workaround would be copying the podspec file to your own project, modifying the s.name from "RainbowSwift" to "Rainbow" and then use your own podspec to integrate:
pod 'Rainbow', :podspec => 'https://your-repo.com/Rainbow.podspec'

source: gharchive/issue

authors: jnewc onevcat

repo: onevcat/Rainbow

license: mit

license_type: permissive

license_source: bigquery


text:
feat!: add select (custom) component

Purpose
Implement non-native Select component.
closes #3
Approach
Reuse Popover for placement techniques, reuse native Select for styling.
Add hooks and conditional logic as React component entry point.
Testing
Preview.
Risks
Changes native Select and Popover styles a bit, but should be retro compatible.

Can you check the form story, I believe something's not right with validation?

Fixed, also refactored for maintainability and improved docs a bit.
Mind checking the newer commits?

I still have to compare both versions and document the breaking changes and migration path.
Pretty sure there's at least one in Select's CSS.
But thanks for the review!

There, some last minute fixes and the migration guide documented.
Feel free to review if you want.
We still need to document the rest of the migration for other breaking changes before we can release 2.0.0.

Awesome, can't wait for v2 release 👏

source: gharchive/pull-request

authors: josokinas rabelloo

repo: onfido/castor

license_source: github-api


text:
Add and check license headers

All source files should have a license header. Add it to all files and add a CI step to ensure we don' forget to add them.

Shall we also upgrade license type here? 🤔
https://github.com/onflow/flow-js-testing/blob/7afda575edeacc858f8b61525360a17bd7be45dc/package.json#L17

@MaxStalker Yes, good catch!

authors: MaxStalker turbolent

repo: onflow/flow-js-testing


text:
NFTStorefront v1 & v2 Cadence 1.0 Refactor

Closes: #87
This PR targets the existing stable-cadence branch, so includes some updates from main. The idea is to use stable-cadence as the working Stable Cadence branch moving forward.
Dependencies pulled from respective repo's stable-cadence branches:

ViewResolver
FungibleToken
NonFungibleToken
MultipleNFT
ExampleNFT

The changes against NFTStorefront and accompanying transactions & scripts have been made to conform to Cadence 1.0 language updates.
Note: It looks like the test suite is running the current version of Cadence as tests are not passing.
TODO:

[x] Merge stable-cadence branch to this one? Picked up from up from #74, so merge might be best way forward.
[ ] Update dependency repos

[ ] flow-ft transactions & scripts on stable cadence feature branch
[ ] flow-nft transactions & scripts on stable cadence feature branch

[ ] Get test cases to pass

Progress on this PR is blocked until flow-ft and flow-nft repos have fully functional stable cadence branches, especially the former.
It appears the transactions from flow-ft are utilized in the test suite and since flow-ft transactions aren't up to Cadence 1.0 in the stable cadence feature branch, they're failing in the NFTStorefront test suite.
Relevant discord discussion: https://discord.com/channels/613813861610684416/621847426201944074/1157425948405280858

@joshuahannan thanks for the comments, newest update should address all of them.
I also removed the go testing entirely which makes the version issue not a problem. The previous go testing was very simple testing targeted only towards storefrontv1, and so I've replaced it with new cadence testing for storefrontv1.

I'm using the stable cadence version of NFTStorefront and borrowNFT is always reverting for me here:
assert(ref.isInstance(self.getDetails().nftType), message: "token has wrong type"), but creating the listing works fine, it would revert on create if type was actually invalid right?
Commenting this line makes it work obviously

I'm using the stable cadence version of NFTStorefront and borrowNFT is always reverting for me here: assert(ref.isInstance(self.getDetails().nftType), message: "token has wrong type"), but creating the listing works fine, it would revert on create if type was actually invalid right?
Commenting this line makes it work obviously

Taking a look at this - In create listing we're using the following code:              assert(nft!.getType() == self.details.nftType, message: "token is not of specified type") which differs from the ref.isInstance(self.getDetails().nftType that's in borrowNFT. I'm not entirely sure why one would work and the other wouldn't, so looking into that.

I'm using the stable cadence version of NFTStorefront and borrowNFT is always reverting for me here: assert(ref.isInstance(self.getDetails().nftType), message: "token has wrong type"), but creating the listing works fine, it would revert on create if type was actually invalid right?
Commenting this line makes it work obviously

Taking a look at this - In create listing we're using the following code:  assert(nft!.getType() == self.details.nftType, message: "token is not of specified type") which differs from the ref.isInstance(self.getDetails().nftType that's in borrowNFT. I'm not entirely sure why one would work and the other wouldn't, so looking into that.

I think a test covering borrowNFT would be sufficient

Thanks for bringing this up.
I added a new script and test case to cover borrowNFT now, also updated the previewnet build to reflect the fix needed to properly check the nft type. If you run into any more issues please keep them coming here!

I updated all the dependencies, addressed all my previous comments on this PR, and got the cadence tests passing. NFTStorefront is staged on testnet, but NFTStorefrontV2 is not staged on testnet because we are still looking for the key information for the account.
Any more reviews here would be much appreciated

@sisyphusSmiling I addressed your comments about the contracts and pushed my changes. I created an issue for the transaction changes that I am going to come back to later

authors: aishairzay highskore joshuahannan sisyphusSmiling

repo: onflow/nft-storefront

license: Unlicense


text:
holder.photoDraweeView.setPhotoUri(uri, mContext); 没有回调图片问题

作者，我需要你这个方法   holder.photoDraweeView.setPhotoUri(uri, mContext); 回调一个 加载成功后的图片资源，我如果我继承你的PhotoDraweeView类，但是 你那个方法里我无法改变你的    private boolean mEnableDraweeMatrix = true; 这个参数，因为它是私有的，如果我直接复制一遍的话，有一个错误就是
@Override protected void onDetachedFromWindow() {
mAttacher.onDetachedFromWindow();
super.onDetachedFromWindow();
}这个方法里的  mAttacher.onDetachedFromWindow();  这个方法是被保护的，不允许我去调用，求解。

难道必须要弄成MODEL 模式 O.O

如果直接用FRECO 自己带的，图片的缩放功能就会失去效果，我看你写了 一个这样的方法，估计就是改那里了

setEnableDraweeMatrix 有这个方法……

嗯 谢谢 我刚才发现了 你写了这个方法

作者 调用这个   public void setPhotoUri(Uri uri, @Nullable Context context)  方法 我加载了 俩张图片  第一张无法 手势缩放，第二张才可以

public void setPhotoUri(Uri uri, @Nullable Context context) {
holder.photoDraweeView.setEnableDraweeMatrix(false);
DraweeController controller = Fresco.newDraweeControllerBuilder()
.setCallerContext(context)
.setUri(uri)
.setOldController(holder.photoDraweeView.getController())
.setControllerListener(new BaseControllerListener() {
@Override
public void onFailure(String id, Throwable throwable) {
super.onFailure(id, throwable);
holder.photoDraweeView.setEnableDraweeMatrix(false);
}
                @Override
                public void onFinalImageSet(String id, ImageInfo imageInfo,
                                            Animatable animatable) {
                    super.onFinalImageSet(id, imageInfo, animatable);
                    holder.photoDraweeView.setEnableDraweeMatrix(true);
                    if (imageInfo != null) {
                        holder.photoDraweeView.update(imageInfo.getWidth(), imageInfo.getHeight());
                    }
                }

                @Override
                public void onIntermediateImageFailed(String id, Throwable throwable) {
                    super.onIntermediateImageFailed(id, throwable);
                    holder.photoDraweeView.setEnableDraweeMatrix(false);
                }

                @Override
                public void onIntermediateImageSet(String id, ImageInfo imageInfo) {
                    super.onIntermediateImageSet(id, imageInfo);
                    holder.photoDraweeView.setEnableDraweeMatrix(true);
                    if (imageInfo != null) {
                        holder.photoDraweeView.update(imageInfo.getWidth(), imageInfo.getHeight());
                    }
                }
            })
            .build();
    holder.photoDraweeView.setController(controller);
}   我写的和你一模一样

貌似   只好 第一行  默认 改成   holder.photoDraweeView.setEnableDraweeMatrix(true); 能缩放了

测试了下 也不好使

这个监听 也失效了 holder.photoDraweeView.setOnPhotoTapListener(this);

用这个方法，手势缩放就失效。
private void setPhotoUri(Uri uri) {
ImageRequest imageRequest = ImageRequestBuilder
.newBuilderWithSource(uri)
.setProgressiveRenderingEnabled(true).build();
Fresco.getImagePipeline().fetchDecodedImage(imageRequest, this).subscribe(new BaseBitmapDataSubscriber() {
@Override
public void onNewResultImpl(@Nullable Bitmap bitmap) {
if (null != bitmap) {
            }
        }

        @Override
        public void onFailureImpl(DataSource dataSource) {

        }
    }, CallerThreadExecutor.getInstance());
    ControllerListener controllerListener = new BaseControllerListener<ImageInfo>() {
        @Override
        public void onFinalImageSet(String id, @Nullable ImageInfo imageInfo, @Nullable Animatable anim) {
            holder.photoDraweeView.setEnableDraweeMatrix(true);
        }

        @Override
        public void onFailure(String id, Throwable throwable) {
            holder.photoDraweeView.setEnableDraweeMatrix(false);
        }
    };
    DraweeController draweeController = Fresco.newDraweeControllerBuilder()
            .setAutoPlayAnimations(true)
            .setImageRequest(imageRequest)
            .setControllerListener(controllerListener)
            .build();
    holder.photoDraweeView.setController(draweeController);
}

如果单独拿出来是可以使用的，可能是你的pager adapter 回收等原因没有处理好 setEnableDraweeMatrix

好的，这个问题解决了，为此贴上源码
public void setPhotoUri(Uri uri) {
ImageRequest imageRequest = ImageRequestBuilder
.newBuilderWithSource(uri).setResizeOptions(new ResizeOptions(400, 400))
.setProgressiveRenderingEnabled(true).build();
ImagePipeline imagePipeline = Fresco.getImagePipeline();
DataSource<CloseableReference> dataSource = imagePipeline.fetchDecodedImage(imageRequest, this);
dataSource.subscribe(new BaseBitmapDataSubscriber() {
@Override
public void onNewResultImpl(@Nullable Bitmap bitmap) {
if (null != bitmap) {
}
}
        @Override
        public void onFailureImpl(DataSource dataSource) {

        }
    }, CallerThreadExecutor.getInstance());
    DraweeController controller = Fresco.newDraweeControllerBuilder()
            .setCallerContext(mContext)
            .setUri(uri)
            .setImageRequest(imageRequest)
            .setOldController(holder.photoDraweeView.getController())
            .setControllerListener(new BaseControllerListener<ImageInfo>() {
                @Override
                public void onFailure(String id, Throwable throwable) {
                    super.onFailure(id, throwable);
                    holder.photoDraweeView.setEnableDraweeMatrix(false);
                }

                @Override
                public void onFinalImageSet(String id, ImageInfo imageInfo,
                                            Animatable animatable) {
                    super.onFinalImageSet(id, imageInfo, animatable);
                    holder.photoDraweeView.setEnableDraweeMatrix(true);
                    if (imageInfo != null) {
                        holder.photoDraweeView.update(imageInfo.getWidth(), imageInfo.getHeight());
                    }
                }

                @Override
                public void onIntermediateImageFailed(String id, Throwable throwable) {
                    super.onIntermediateImageFailed(id, throwable);
                    holder.photoDraweeView.setEnableDraweeMatrix(false);
                }

                @Override
                public void onIntermediateImageSet(String id, ImageInfo imageInfo) {
                    super.onIntermediateImageSet(id, imageInfo);
                    holder.photoDraweeView.setEnableDraweeMatrix(true);
                    if (imageInfo != null) {
                        holder.photoDraweeView.update(imageInfo.getWidth(), imageInfo.getHeight());
                    }
                }
            })
            .build();
    holder.photoDraweeView.setController(controller);
    imagePipeline.evictFromDiskCache(uri);

}

authors: ongakuer xiedong

repo: ongakuer/PhotoDraweeView


text:
opening a big file

Hello,
I have some log files that are between 4 and 5 millions line long. I usually open them with VSCode. It takes about 5-6 secondes to load, but once it is there, scrolling through it is quite smooth.
I tried opening one of them with oni2, and it opens instantly (congrats on that !!). But then, scrolling through it is extremely slow, to the point that it is unusable..
Just thought I'd drop an issue, maybe there's something blocking the smooth scrolling of long files that could be fixed.

Thanks for the report, @giltho !
I suspect that it might actually be an issue we're hitting with length of lines - I was just experimenting with some files:

sqlite3.c - a giant C file with around ~250k lines
canada.json - a large JSON file with only 7 lines, but some very long lines

It turns out we are having issues scrolling through the canada.json - due to the long lines - whereas the sqlite3.c scrolls pretty smooth.
Just some questions though to help narrow down:

Which OS is this?
Are you scrolling via the keyboard (j/k/<C-d>/<C-u>) or via the mouse (scrollwheel / minimap)
Is it possible to share the log files for examination (to help confirm if it is 'line length' vs 'number of lines'). Of course, don't worry if there is any chance of sensitive info there!

Thanks for reporting the issue!

Happy to help, even just a tiny bit :)
On MacOS Mojave 10.14.5
I was scrolling using the pad (scrollwheel), but it's also slow to just move in any direction with j/k
The log file is far from being sensitive, but it is a bit heavy (187MB) (containing ~4M7 lines), I don't really know how to send it. I'll come back to you on that.
However, the longest line is less than 200 characters long, so I believe it would disprove your theory.
Also, while scrolling it, I got extremely weird behaviour from oni. I attached a video because it is very weird.

first I started scrolling and then, the cursor jumped back up, and then it just moved up and down a line on its own (I wasn't touching the keyboard at all at that moment).
I tried closing the file, but it didn't work, instead the order of tabs just started rotating.
I don't know if it is related to the size of the file.

(the video was a bit slowed in the conversion from mov to gif, it was jumping a bit faster than that)

I think there even might have been a regression lately, around the commits in https://github.com/onivim/oni2/issues/1670.
While the search is better on large files - once they open, but thanks nevertheless - they open real slow, and eat around a GB of memory for a 10 MB file

Hi @Samonitari ,
Do you have an example of a file that's opening slowly? A couple possible culprits I can think of:

A regression around sending the data to the extension host (and for very large files, we shouldn't be sending them at all..)
A regression around syntax highlighting
A regression around font fallback (loading excessive font faces / glyphs)

Even the type of file would be helpful, and if you have any extensions installed that work with it.

SubSystem172.31.2.60.log
It is a harmless log file.
The custom extensions installed, (I removed some that are more heavyweight, but the issue persists)

Better TOML
Git Histroy
Horizon Theme (not active)
Python
PowerShell Preview
YAML
Windows Bat Language Basics
Git
Powershell Language Basics
Rust Language Basics
XML Language Basics

Well, maybe they don't work all (I did not test them all throroughly), but none of them should be actively doing anything on a log file. File is neither in a git repo, nor in a project folder of a programming language.
And there is an interesting unrelated bug I just found on this file:
I tried to make it bigger, (as the actual 10MB log of a different subystem contains confidential infos, I grabbed this harmless one). Done: Visual select from first to last line, yank, paste -> pasted lines will have alternating empty lines.
Without OCaml/Reason/Javascript knowledge, I cannot contrib any code, but some occasional real-life stress I can gladly do XD

Excuse me for the question, but are there any news on this?
BTW, I observed something interesting:
If you open the same 10MB file on Windows through the system wise registered context menu "Open with Onivim 2" it will hog even more RAM, around 1.8 GB!
Opening from an already running Onivim 2 takes around 1 GB, as I mentioned.
I used a clear install (removed AppData folder after uninstall), without any extensions
It can be a .log or .txt file (it doesn't matter).

authors: Samonitari bryphe giltho

license: MIT


text:
fix(ux/#2404): support macos shortcuts in inputs

Issue: Shortcuts in the text inputs are not handled on macOS.
Fix: Added handling for most common macOS keyboard shortcuts.
NOTE: While testing all the keyboard shortcuts by hand I noticed that for some reason <C-w>, <C-LEFT> and <C-RIGHT> don't do anything on my mac. It's quite mysterious since <C-h> and <S-C-LEFT> for example work just fine. These issues were repeatable on the master branch as well, so not a result of this PR. @bryphe perhaps you could point me to where the keyboard input handling happens - I could do some digging and see if I can figure out why those shortcuts don't work my end?

Hi @fiddler - thanks for adding these! Very nice improvement to the text input on OSX.

While testing all the keyboard shortcuts by hand I noticed that for some reason ,  and  don't do anything on my mac. It's quite mysterious since  and  for example work just fine.

Good question, thanks for the thorough testing. It looks like two different issues here -
Issue 1: <C-LEFT> & <C-RIGHT> not working
This is due to OSX assigning these keybindings to mission control / change workspaces.
You can check if these keybindings are set by going to System Preferences -> Keyboard -> Shortcuts -> Mission Control (they're set by default, I believe):

If these shortcuts are set - the key presses don't even make it to Onivim to process. I turned off these shortcuts, and I saw <C-LEFT> and <C-RIGHT> input be processed after that.
Issue 2: <C-w> not working
The <C-w> issue is because, in some parts of the UI with text inputs, we allow window navigation - so the <C-w> is being picked up by bindings like <C-w><C-j> to move to a different window.
There's a couple of possibilities - discussed a bit here too: https://github.com/onivim/oni2/pull/2518 - we either can disable the window-navigation bindings in text input, and set it up the input text to be 'modal' (ie, you'd need to press <esc> to enter normal mode, and then use window navigation) - or we could have an alternate key for initiating windows navigation (like the termwinkey setting in vim). I need to think a bit more about the user experience around that. Let me know if you have ideas.

These issues were repeatable on the master branch as well, so not a result of this PR.

Definitely not related to your PR 👍  We can go ahead and merge this, and leave in the <C-w> binding - and depending on the solution for issue 2, it'll eventually be picked up.

Thanks for the in-depth explanations @bryphe!
Disabling those Mission Control shortcuts indeed made the <C-LEFT/RIGHT> work. For the <C-w> handling I do not have an informed opinion to offer since I've not used the window navigation bindings.
Seems there is a bunch of checks failing - is there anything I should do regarding this PR to get it to pass? I checked some of the errors, but not really sure what those are about.

/azp run

Looks like CI is green now - thanks @CrossR for kicking off a new build, and thanks @fiddler for the contribution!

authors: CrossR bryphe fiddler


text:
[WIP] Migration of 1.x to 2.x (preview) of existing WPF library

Currently migrating an existing component on a specific branch, see https://github.com/wildgums/orc.filterbuilder/tree/feature/netcore30
Will post the progress here, it might help iron the last bits out / maybe help others hitting the same bumps.

Upgrade to 2.0.0-preview.7 in global.json
Add netcoreapp3.0 (note that you don't have to include the frameworkreference, it will be done by msbuild sdk extras)
Make sure to enable this in the project settings:

true

This works fine for non-xaml related libraries (outputting a netcoreapp3.0 library). Hitting this error:
2>C:\Source\Orc.FilterBuilder\src\Orc.FilterBuilder.Xaml\Orc.FilterBuilder.Xaml.csproj : error NU1105: Unable to find project information for 'C:\Source\Orc.FilterBuilder\src\Orc.FilterBuilder\Orc.FilterBuilder.csproj'. Inside Visual Studio, this may be because the project is unloaded or not part of current solution. Otherwise the project file may be invalid or missing targets required for restore.
2>Done building project "Orc.FilterBuilder.Xaml.csproj" -- FAILED.

Do you have a repro? Can you remove the bin\obj directories and make sure to run msbuild /restore to make sure it restores the new bits?

It's a separate branch on the repository (feature/netcore30), but I am updating my sdk's first, could be that. Will try to keep this thread up-to-date while I am investigating.

Keep in mind that for a pure WPF library, you shouldn't need the extras anymore (though it should still work as it delegates to the built-in SDK).
It should also fall back to existing current behavior if the nightly SDK isn't there, but it may not be setting ExtrasEnableWpfProjectSetup

Note: make sure to update the SDK, it was not working for me until I updated to the latest (one before was 1 week old):

Looks like most of it is working, last one left, but still investigating:
2>C:\Source\Orc.FilterBuilder\src\Orc.FilterBuilder.Xaml\Views\EditFilterView.xaml(54,10): error MC3074: The tag 'Interaction.Behaviors' does not exist in XML namespace 'http://schemas.microsoft.com/expression/2010/interactivity'. Line 54 Position 10.

This is because a dependency explicitly specifies it for a target framework, will probably need to reference it manually for now.

@onovotny Question: can you safely use UseWpf on a multi-targeting lib targeting other stuff (xamarin, uap) or should I protect like this:
  <!-- .NET 4.6 -->
  <PropertyGroup Condition="'$(TargetFramework)' == 'net46'">
    <UseWpf>true</UseWpf>
  </PropertyGroup>
  <ItemGroup Condition="'$(TargetFramework)' == 'net46'">
    <PackageReference Include="Expression.Blend.Sdk.WPF" Version="1.0.1" />
    <Reference Include="System.Windows.Forms" />
  </ItemGroup>

Looks like Orc.FilterBuilder is compiling as .NET Core 3.0 lib with the example app also being a .NET Core 3.0 app :-)
Fantastic work @onovotny , thanks for all the effort you are putting into this library.

I would condition UseWpf on the TFM's you want as the WindowsDesktop SDK sets default globs which may conflict.

You also don't need to include System.Windows.Forms on .NET 4.6, just set UseWindowsForms to true as well.

Thanks for the tip, will do that instead. I think that initially came from before I was using the extras.

While converting another project, I was able to build using VS 2017 preview, but not using VS 2017. Trying to install the x86 SDK side-by-side with the X64 one to see if that solves the issue.

Found something in the release notes:

.NET Core Tools for Visual Studio
Starting with this release, the .NET Core tools for Visual Studio will now default to using only the latest stable version of a .NET Core SDK that is installed on your machine for GA releases of Visual Studio. For future previews, the tools will use only preview .NET Core SDKs.

authors: GeertvanHorrik onovotny

repo: onovotny/MSBuildSdkExtras


text:
add compact DB mode (--compact-db) to de-duplicate mbtiles output

Context
this addresses point 2 of https://github.com/onthegomap/planetiler/issues/167#issuecomment-1115483329
Overview
the compact DB mode splits the tiles table into tiles_shallow and tiles_data

tiles_shallow contains the coordinates plus a reference on the data ID
tiles_data contains the data ID plus the actual tile data

this allows to de-duplicate content since multiple tiles can reference the same data
in this mode, tiles is realized as a view that joins the two tables tiles_shallow and tiles_data
Impact

it can significantly de-crease the mbtiles file size iff there's many duplicates (ocean tiles) - in case of Australia: 3.1GB -> 1.7GB
it has little to no impact on writing mbtiles (if no dupes it's slower, and the more dupes the faster but not significantly)
reading from the mbtiles is 10-15% slower because of the indirection over the view

some more insights can be found here: PR-impact.txt
Concept
The main idea to realize this is to create a hash over the tile data, and build a map between that hash and a data ID.
Failed Approaches
I was a bit surprised that the impact on writing the mbtiles is little to none. The only explanation I have is that we have to do more inserts than before (tiles_shallow, tiles_data). I tried to offload this "dual insert" from code to the DB. So in planetiler I produced just one insert but the insert was into a view - which then either just inserted into tiles_shallow or both tiles_shallow and tiles_data. This, however, increased the overall mbtiles generation time by almost 20%.
Next
I'd like to address point 1 in https://github.com/onthegomap/planetiler/issues/167#issuecomment-1115483329  either in this or a follow-up PR

Would using a faster non-cryptographic hashing algorithm improve performance much? Or is performance so limited by IO that it doesn't matter?

I got around to testing this using australia-oceania as the region.
Here are the file size differences:
-rw-rw-r--  1 xxxx xxxx 4.9G May 10 14:56 compact.mbtiles
-rw-rw-r--  1 xxxx xxxx 2.3G May 10 14:56 compact.mbtiles.zst
-rw-rw-r--  1 xxxx xxxx  12G May 10 15:16 full.mbtiles
-rw-rw-r--  1 xxxx xxxx 2.4G May 10 15:16 full.mbtiles.zst

When uncompressed, compacted mbtiles are a LOT smaller. There isn't much difference when compressing the files with zstandard, which is probably to be expected.
I tried serving the tiles with tileserver-gl, and while I have no performance figures, the compacted tiles do feel slightly slower than the full tileset. But by the time you add some tile caching into the mix, it may not matter much.

Thanks for this change @bbilger! This looks like a huge improvement

Would using a faster non-cryptographic hashing algorithm improve performance much? Or is performance so limited by IO that it doesn't matter?

Have you tried samplig with visualvm to see how much of a hot spot the hash is?
I'm also looking into generating pmtiles output, and it uses the FNV hash to solve this problem.  I haven't checked for collisions but with a 64 bit hash over 200-300 million tiles it should theoretically be a 1 or 2 out of 1000 chance of a collision, bit it is very simple and fast:
  private static final long FNV1_64_INIT = 0xcbf29ce484222325L;
  private static final long FNV1_PRIME_64 = 1099511628211L;
  public static long hash(byte[] data) {
    long hash = FNV1_64_INIT;
    for (byte datum : data) {
      hash ^= (datum & 0xff);
      hash *= FNV1_PRIME_64;
    }
    return hash;
  }

Maybe @bdon has a better sense of fnv hash's suitability for planet-scale tilesets?

I guess at the scales we're talking about (millions of tiles), one would really want the chance of collisions to be so low, that you don't have to worry about checking for collisions, otherwise I think you'd have to do a binary compare to know for certain. The last thing we would want for a data tile to collide with an ocean tile and not get included...
I think you end up going with a faster hash algorithm, a 128 bit hash would probably be the safer option, and I imagine it should still be faster than sha1/md5.

@shermp
I wrote "com.onthegomap.planetiler.mbtiles.Compare" to compare reading from equal mbtiles files (e.g. compact vs. non-compact) and to get some idea about the performance decrease. On my machine, reading from australia-compact is 10-15% slower than from australia-not-compact.
@msbarry / @shermp
What I did so far was swapping SHA-1 by Murmur3 128 - and it had zero impact on the performance which made me believe this is not the bottleneck. But all I did was checking the stats produced by planetiler at the end, and haven't analyzed it any further but will try to do so now.
@msbarry
re: 1.: The threshold sounds like a good idea. Do you want it to be configurable as CLI argument?
re 2.: Do I understand the idea correctly that you effectively want to hash twice - in case of a duplicate? First layer would hash into a small space and store the result to a BitSet, and the second layer would use a "real" hash function (e.g. FNV 32) and store the result in a HashMap (hash, dataId). Is the idea here (in case of no duplicate) that the first hashing would be even cheaper, and looking up from a BitSet is a bit faster than from a HashMap? Sorry, just want to double-check that I get the idea, since it adds some complexity and FNV-32 already looks pretty fast.

I tried to address your feedback in https://github.com/onthegomap/planetiler/pull/219/commits/0faa63aad43eb62d8cf3dcb4e86323abc9d544a3 => please let me know what you think.
Testing with Australia, I can barely notice any difference but it seems a tiny bit faster. As I said before, I don't think the hashing is the bottleneck and at least I cannot identify it as one in VisualVM but it's probably the right thing to do to not use SHA-1.
One reason I decided to use SHA-1 in the beginning was that I wanted to avoid any collisions. As such I share @shermp 's concerns that using FNV 32 might be a bit problematic since a collision would be rather bad. Maybe it would be better to use FNV 64, FNV 128, or Murmur3-128?

re: 1.: The threshold sounds like a good idea. Do you want it to be configurable as CLI argument?

Yes, if we go with a size-based approach then configurable would be good, probably with a default of ~20kb.

re 2.: Do I understand the idea correctly that you effectively want to hash twice - in case of a duplicate? First layer would hash into a small space and store the result to a BitSet, and the second layer would use a "real" hash function (e.g. FNV 32) and store the result in a HashMap (hash, dataId). Is the idea here (in case of no duplicate) that the first hashing would be even cheaper, and looking up from a BitSet is a bit faster than from a HashMap? Sorry, just want to double-check that I get the idea, since it adds some complexity and FNV-32 already looks pretty fast.

The downside to option 1 is that a repeated fill tile with lots of attributes could get above 20kb, so option 2 is an adaptive way to identify tiles we might have seen before. On second thought, we'd probably want to use a bloom filter for that, so it's probably simpler just to use a size threshold.

Instead of using a size threshold, could we limit the deduplication only to tiles that have a single layer, or a single feature on a single layer? We know a priori that the deduplication is useful for ocean squares and land squares, which might have arbitrary tags that increase the total tile size. We would need the ability to introspect on the PBF content instead of treating it as a blob. Are there good counterexamples on why this won't work?

It is possible that we can pass through an "is fill" bit for each feature, then inspect easily if all features in a tile are fill polygons without deserializing - that's what I was planning for #168.
So maybe to start on this PR, we could limit the deduping to just tiles with less than some small number of features (1? 2? 5?) and do the more robust fill identification as part of #168 ?  There's a getNumFeaturesToEmit method to get the number of features in TileFeatures argument to the tile encoder without deserializing anything.

"NumFeaturesToEmit" seems like the right thing to do and I implemented it accordingly.
Duplicates in Australia where spread as follows per "NumFeaturesToEmit":
{1=13074826, 2=1359105, 3=397821, 4=13822, 5=73, 7=4, 8=2, 9=2, 18=2}
=> trying to de-duplicate/hash only if less than 5 "NumFeaturesToEmit"
...
gonna tackle packing tiles_shallow x,y,z into one int in the DB now next

Ran some numbers today, looks like it's possible to encode tiles up to zoom level 29 in a 64 bit int.
Considering that at each zoom level, each axis has 2zoom tiles, you can encode tiles such:
Bits 0-4 could be zoom
Bits 5-33 could be y
bits 34-62 could be x
Or any other similar scheme.

:+1:
Someone please correct me if I am wrong but I think the max supported zoom level is 14. This - I think - allows to pack into a 32bit integer.
For "packing" x,y,z into on int, I am using
TileCoord#encode
and on the generated columns, I am currently experimenting with, I am basically using TileCoord#decode

Are you generating the x,y columns as part of the SQL view?

Well, yes but mostly no. Technically the view remains unchanged to what it is now for the compact mode:
CREATE VIEW tiles AS
select
  tiles_shallow.zoom_level as zoom_level,
  tiles_shallow.tile_column as tile_column,
  tiles_shallow.tile_row as tile_row,
  tiles_data.tile_data as tile_data
from tiles_shallow
join tiles_data on tiles_shallow.tile_data_id = tiles_data.tile_data_id

The key change is to have generated and most importantly indexed columns in tiles_shallow:
CREATE TABLE tiles_shallow (
  tile_id integer,
  tile_data_id integer,

  zoom_level integer as ( (tile_id >> 28) + 8 ),
  tile_column integer as ( (tile_id >> 14) & ((1 << 14) - 1) ),
  tile_row integer as ( (1 << zoom_level) - 1 - (((1 << zoom_level) - 1) - ((tile_id) & ((1 << 14) - 1))) )
)
CREATE UNIQUE INDEX tiles_shallow_index on tiles_shallow (zoom_level, tile_column, tile_row)

With a view-only approach like the following we would end up with a full table scan which would hurt read performance badly
CREATE VIEW tiles AS
select
  ( (tiles_shallow.tile_id >> 28) + 8 ) as zoom_level,
  ( (tiles_shallow.tile_id >> 14) & ((1 << 14) - 1) ) as tile_column,
   ( (1 << ( (tiles_shallow.tile_id >> 28) + 8 )) - 1 - (((1 << ( (tiles_shallow.tile_id >> 28) + 8 )) - 1) - ((tile_id) & ((1 << 14) - 1))) ) as tile_row,
  tiles_data.tile_data as tile_data
from tiles_shallow
join tiles_data on tiles_shallow.tile_data_id = tiles_data.tile_data_id

What I can say so far is that:

write performance is mostly unaffected - maybe a slight improvement
file size is slightly lower
haven't tested impact on read yet

:+1:
Someone please correct me if I am wrong but I think the max supported zoom level by planetiler is 14. This allows to pack it into a 32bit integer.
For now yes although it will increase to 15 or 16 at some point in the future.

Someone please correct me if I am wrong but I think the max supported zoom level by planetiler is 14. This allows to pack it into a 32bit integer.

For now yes although it will increase to 15 or 16 at some point in the future.

write performance is mostly unaffected - maybe a slight improvement

+1 I would expect some write performance increase to come from a simpler index structure, but most to come from being able to fit more tiles into each batch with the 999 limit on prepared statement params
Curious how read performance turns out. To query from application layer, it would be relatively efficient to map from x/y/z to tile ID, then lookup the data for it but I'm not sure how possible it is to express that in a sqlite view...

I cannot see a noticeable impact on read performance (BenchmarkMbtilesRead shows something like 1% faster).
The approach - let's call it approach no.1 -  presented here https://github.com/onthegomap/planetiler/pull/219#issuecomment-1124366496 relies on generated x,y,z columns that need to be indexed. Thus the index structure is/should be (mostly) identical.
Unless you want to build a sqlite file that breaks with mbtiles spec, approach no.1 is the only feasible option I can see.
A full table-scan - let's call it approach no.2 - as described at the end of https://github.com/onthegomap/planetiler/pull/219#issuecomment-1124366496 is obviously a no-go.
This bring us to approach no.3:
create view tiles as
  select zoom_level, tile_column, tile_row, tile_data
  from tiles_shallow ts
  join tiles_data on ts.tile_data_id = tiles_data.tile_data_id
  left join zoom_levels
  left join tile_columns
  left join tile_rows
  where ts.tile_id = ((zoom_level-8) << 28) | (tile_column << 14) | (tile_row);

(bear with me if the formula is not quite correct, haven't tested it much but it worked at least for my test coordinates)
zoom_levels, tile_columns, tile_rows are views on a number table (create table numbers (number integer primary key) and it contains numbers from 0 to 1 << 14). I just created it manually but while it's relatively quick (~5ms), it's definitely slower, and it breaks pretty quickly if you don't filter it completely with zoom_level, tile_column, tile_row.

Someone please correct me if I am wrong but I think the max supported zoom level by planetiler is 14. This allows to pack it into a 32bit integer.

For now yes although it will increase to 15 or 16 at some point in the future.

write performance is mostly unaffected - maybe a slight improvement

+1 I would expect some write performance increase to come from a simpler index structure, but most to come from being able to fit more tiles into each batch with the 999 limit on prepared statement params
Curious how read performance turns out. To query from application layer, it would be relatively efficient to map from x/y/z to tile ID, then lookup the data for it but I'm not sure how possible it is to express that in a sqlite view...

I cannot see a noticeable impact on read performance (BenchmarkMbtilesRead shows something like 1% faster).

Cool, I would expect the fastest one to be the original, so mostly interested how much read qps drops with the more compact formats. If we can get a compact format with same read performance as original that would be ideal.

The approach - let's call it approach no.1 - presented here #219 (comment) relies on generated x,y,z columns that need to be indexed. Thus the index structure is/should be (mostly) identical.

OK, is there any benefit of this one then? It seems the same as x/y/z columns, but with some added complexity?

Unless you want to build a sqlite file that breaks with mbtiles spec, approach no.1 is the only feasible option I can see. A full table-scan - let's call it approach no.2 - as described at the end of #219 (comment) is obviously a no-go.
This bring us to approach no.3:
create view tiles as
  select zoom_level, tile_column, tile_row, tile_data
  from tiles_shallow ts
  join tiles_data on ts.tile_data_id = tiles_data.tile_data_id
  left join zoom_levels
  left join tile_columns
  left join tile_rows
  where ts.tile_id = ((zoom_level-8) << 28) | (tile_column << 14) | (tile_row);

(bear with me if the formula is not quite correct, haven't tested it much but it worked at least for my test coordinates) zoom_levels, tile_columns, tile_rows are views on a number table (create table numbers (number integer primary key) and it contains numbers from 0 to 1 << 14). The number table is a trick to get hold of the requested parameters zoom_level, tile_column, tile_row to reconstruct the tile_id. I just created it manually but while it's relatively quick (~5ms), it's definitely slower, and it breaks pretty quickly if you don't filter it completely with zoom_level, tile_column, tile_row.

OK cool thanks for exploring this, but if it's looking like there might not a good way to express this through sqlite views then feel free to ignore!

Also @bbilger let me know if any of those need more clarification, or if you'd rather have me push the change for one to your branch!

@msbarry all is clear. Sorry, I just didn't have to work on it over the weekend. I should have addressed the majority of your comments absolutely latest by Friday.

OK, is there any benefit of this one then? It seems the same as x/y/z columns, but with some added complexity?

The advantage is that write performance increases since the prepared statement will have just 1 vs 3 arguments. Let me re-write the write benchmark then a decision can be based on that.

Sorry for the delay, this looks great! Leaving a few minor comments, but I see no major issues remaining before we merge. Thanks for putting this together and improving the code long the way!

Absolutely no problem. Thank you for the review and the feedback. I tried to incorporate your feedback in this commit https://github.com/onthegomap/planetiler/pull/219/commits/c65c7b7d23007973e0d4cf8cab4ed673d91be3d9 . Let me know if there's anything else.

Ran some tests this morning on the planet with this branch merged with #225.  --compact-db appears to improve write throughput by 5%, hurt read throughput by 5%, and make the mbtiles file about 25% smaller.
--compact-db=false:
0:38:41 INF - 	mbtiles          14m13s cpu:5h45m29s gc:45s avg:24.3
0:38:41 INF - 	  read    2x(24% 3m22s sys:58s wait:8m20s done:28s)
0:38:41 INF - 	  merge   1x(35% 5m3s sys:13s wait:8m26s done:13s)
0:38:41 INF - 	  encode 62x(33% 4m38s sys:8s wait:8m53s done:13s)
0:38:41 INF - 	  write   1x(87% 12m25s sys:1m29s wait:55s)
0:38:41 INF - ----------------------------------------
0:38:41 INF - 	features	184GB
0:38:41 INF - 	mbtiles	104GB

--compact-db=true:
0:37:38 INF - 	mbtiles          13m29s cpu:5h50m42s gc:46s avg:26
0:37:38 INF - 	  read    2x(26% 3m29s sys:1m1s wait:7m18s done:25s)
0:37:38 INF - 	  merge   1x(32% 4m22s sys:14s wait:8m23s done:11s)
0:37:38 INF - 	  encode 62x(35% 4m44s sys:9s wait:8m2s done:11s)
0:37:38 INF - 	  write   1x(87% 11m40s sys:1m18s wait:53s)
0:37:38 INF - ----------------------------------------
0:37:38 INF - 	features	184GB
0:37:38 INF - 	mbtiles	84GB

The read benchmark shows about 67.7k reads/s with regular and 64.7k reads/s with compact (I was running short on time so did a smaller test)
0:00:00 DEB - argument: bench_repetitions=3 (number of repetitions)
0:00:00 DEB - argument: bench_nr_tile_reads=50000 (number of tiles to read)
0:00:00 DEB - argument: bench_pre_warms=1 (number of pre warm runs)
0:00:00 DEB - argument: bench_mbtiles=data/output.mbtiles,data/compact.mbtiles (the mbtiles file to read from)
0:00:36 INF - working on data/output.mbtiles
0:00:43 INF - readOperationsPerSecondStats: DoubleSummaryStatistics{count=3, sum=203141.152027, min=62994.850298, average=67713.717342, max=70282.853583}
0:00:43 INF - working on data/compact.mbtiles
0:00:51 INF - readOperationsPerSecondStats: DoubleSummaryStatistics{count=3, sum=193959.728756, min=63488.766430, average=64653.242919, max=65851.192021}
0:00:51 INF - diffs
0:00:51 INF - "data/compact.mbtiles" to "data/output.mbtiles": avg read operations per second improved by 4.73367504185164%

Previous tests before this change took about 12m +/- 10s to write mbtiles so I don't have any concerns about --compact-db=false performance being different from what's on main now.

authors: bbilger bdon msbarry shermp

repo: onthegomap/planetiler


text:
support Protege 5.5

The stable version of Protege 5.5 is out. We should support it.

closed by 0ddf7a7154fc262d23fa3eb29e75b149e89b57ba

repo: ontop/ontop


text:
Hermes Engine dateFormat support

Overview
Hermes currently supports toLocaleTimeString and toLocaleString on Android only.  As a result, time is often the wrong timezone and the format of the date for local may be incorrect on iOS.
https://github.com/onubo/react-native-logs/blob/9dfc362ec7a18885f9ad4844f69ac63e877086e2/src/index.ts#L295-L303
Proposal

Add alternative implementations of these formatters AND/OR
Allow dateFormat to be a function so that the user can control formatting

import {format} from 'date-fns'
import { logger, consoleTransport } from "react-native-logs";

const config = {
  dateFormat: (date) => format(date, 'h:mm a')
}

const log = logger.createLogger(config);

Added in v 5.0.0

repo: onubo/react-native-logs


text:
Formatting Improvements

Need to merge fixed formatting with the new changes in master

I merge the master into this branch. Whenever there was a conflict between the master and the fixed_formatting branch, I replaced the code of the fixed_formatting branch by the code of the master branch. This should ensure that no functionality of the master branch will be compromised. However, we have now a bunch of code style vialotations in the hydrophone_request and and Noise_Interferometry module.

I think the most style violations are just missing white spaces.

In the last push I also excluded E999 from the style checker. This rules looks for syntax errors, but apparently it doesn't understand formatted string literals (I think these are new with Python 3), which we used in some print statements.

repo: ooipy/ooipy


text:
Flip the backend for one day to use the prioritised lists

Make a flip of the backend for one day on the existing API endpoint and measure how well it will work

This has been done.

repo: ooni/backend


text:
Set Logging Level

What does SetLevel() do?
If I set the logging level to logging.INFO, will only the message written to logging.Info() be written?
Sorry for the remedial question.
Thank you

Yes, you need to pass the constant (logging.DEBUG / logging.INFO) and the name of your logger instance. Only messages written to the logger at the specified allowed level will then be output, others will be discarded.
e.g. use logging.SetLevel(logging.INFO, "myLog")
logger.Info("my message") > is output
logger.debug("my message") !output

Dosent work
logging.SetLevel(logging.INFO,"")
	backend := logging.NewLogBackend(os.Stderr, "", 0)
	backend2Formatter := logging.NewBackendFormatter(backend, format)

	logging.SetLevel(logging.INFO, "myLog")

	if strings.ToUpper(*LOG_LEVEL) == "ERROR" {
	 	logging.SetLevel(logging.ERROR, "myLog")
	} else if strings.ToUpper(*LOG_LEVEL) == "INFO" {
	    logging.SetLevel(logging.INFO, log.Module)
	} else if strings.ToUpper(*LOG_LEVEL) == "DEBUG" {
		logging.SetLevel(logging.DEBUG, "myLog")
	} else if strings.ToUpper(*LOG_LEVEL) == "CRITICAL" {
		logging.SetLevel(logging.CRITICAL, "myLog")
	} else if strings.ToUpper(*LOG_LEVEL) == "NOTICE" {
		logging.SetLevel(logging.NOTICE, "myLog")
	} else if strings.ToUpper(*LOG_LEVEL) == "WARNING" {
		logging.SetLevel(logging.WARNING, "myLog")
	} else {
		logging.SetLevel(logging.INFO, log.Module)
	}

	logging.SetBackend(backend2Formatter)
	log.Info(logging.GetLevel(log.Module))

Try this
stdout := logging.NewLogBackend(os.Stdout, "", 0)

format := logging.MustStringFormatter(`%{color}%{time:15:04:05.000} %{shortfunc} ▶ %{level:.5s} %{id:03x}%{color:reset} %{message}`)
logging.SetFormatter(format)

levelBackend := logging.AddModuleLevel(stdout)
switch strings.ToUpper(*LOG_LEVEL) {
case "CRITICAL":
	levelBackend.SetLevel(logging.CRITICAL, "")
case "ERROR":
	levelBackend.SetLevel(logging.ERROR, "")
case "WARN":
	levelBackend.SetLevel(logging.WARNING, "")
case "NOTICE":
	levelBackend.SetLevel(logging.NOTICE, "")
case "DEBUG":
	levelBackend.SetLevel(logging.DEBUG, "")
default:
	levelBackend.SetLevel(logging.INFO, "")
}

logging.SetBackend(levelBackend)

@utkudarilmaz Thanx!

Hi! can i change the log level after the log was initialized? i see that SetLevel() doesn't have any locking mechanism. is this the right way to do it? Logging.SetLevel()? or init the log again?
Thanks.

authors: adibraver baloo32 fixNoobi rdhillbb utkudarilmaz

repo: op/go-logging


text:
Crash

After updating libundirect to 1.1, springboard crashed and crashlog says it's because of EmojiPort i think.
CrashLog: https://controlc.com/00f61191
https://cdn.discordapp.com/attachments/570634688813858820/820301565209608233/image0.png
iPhone 5S, iOS 12.5.1, Chimera

EmojiAttributes needs to be updated support libundirect 1.1


text:
[Agent] comps/agent /langchain/ validate service fails

Follow agent comp started the TGI and Agent service, agent service failed.
start agent microservices with tgi
TGI service works, however, validate agent service failed.
Set the Environment
export ip_address=$(hostname -I | awk '{print $1}')
export model="Intel/neural-chat-7b-v3-3"
export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
export HF_TOKEN=${HUGGINGFACEHUB_API_TOKEN}

start TGI service (with No Guadi device, run TGI on Xeon):
docker run -d -it --name "comps-tgi-xeon-service" -p 8080:80 -v ./data:/data -e HF_TOKEN=$HF_TOKEN -e  HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN} -e http_proxy=${http_proxy} -e https_proxy=${https_proxy} -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host -e no_proxy=${no_proxy}  ghcr.io/huggingface/text-generation-inference:sha-e4201f4-intel-cpu --model-id $model --max-input-tokens 4096 --max-total-tokens 8092  --cuda-graphs 0

1.1 validate TGI service works
 curl http://${ip_address}:8080/generate \
      -X POST \
      -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":17, "do_sample": true}}' \
      -H 'Content-Type: application/json'

output correct:
{"generated_text":"\n\nDeep Learning is a subset of Machine Learning. It basically builds algorithms that try"}

Start agent service
docker image: opea/agent-langchain:latest (v1.0 pull from docker hub)

# Agent
docker run -d --runtime=runc --name="comps-langchain-agent-endpoint" -v $WORKPATH/comps/agent/langchain/tools:/home/user/comps/agent/langchain/tools -p 9090:9090 --ipc=host -e HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN} -e model=${model} -e ip_address=${ip_address} -e strategy=react_langchain -e llm_endpoint_url=http://${ip_address}:8080 -e llm_engine=tgi -e recursion_limit=5 -e require_human_feedback=false -e tools=/home/user/comps/agent/langchain/tools/custom_tools.yaml opea/agent-langchain:latest

check status
docker logs comps-langchain-agent-endpoint
docker logs comps-langchain-agent-endpoint
/home/user/.local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:132: UserWarning: Field "model_name_or_path" in Audio2TextDoc has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
[2024-10-21 04:00:37,249] [    INFO] - Base service - CORS is enabled.
[2024-10-21 04:00:37,250] [    INFO] - Base service - Setting up HTTP server
[2024-10-21 04:00:37,250] [    INFO] - Base service - Uvicorn server setup on port 9090
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:9090 (Press CTRL+C to quit)
[2024-10-21 04:00:37,258] [    INFO] - Base service - HTTP server setup successful

The following Validate Microservice fails
curl http://${ip_address}:9090/v1/chat/completions -X POST -H "Content-Type: application/json" -d '{
     "query": "What is the weather today in Austin?"
    }'

expected output
data: 'The temperature in Austin today is 78°F.</s>'

data: [DONE]

Actual result:
Note
HUGGINGFACEHUB_API_TOKEN is set and used in first TGI services. But the log to check HUGGINGFACEHUB_API_TOKEN.
(2024-10-21T04:10:05.346595464Z pydantic_core._pydantic_core.ValidationError: 1 validation error for HuggingFaceEndpoint 2024-10-21T04:10:05.346602020Z   Value error, Could not authenticate with huggingface_hub. Please check your API token. [type=value_error, input_value={'endpoint_url': 'http://...://10.239.182.158:8080'}, input_type=dict] 2024-10-21T04:10:05.346611397Z     For further information visit https://errors.pydantic.dev/2.9/v/value_error )
2024-10-21T04:00:36.995669826Z /home/user/.local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:132: UserWarning: Field "model_name_or_path" in Audio2TextDoc has conflict with protected namespace "model_".
2024-10-21T04:00:36.995795632Z
2024-10-21T04:00:36.995803681Z You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
2024-10-21T04:00:36.995808817Z   warnings.warn(
2024-10-21T04:00:37.250048725Z [2024-10-21 04:00:37,249] [    INFO] - Base service - CORS is enabled.
2024-10-21T04:00:37.250983665Z [2024-10-21 04:00:37,250] [    INFO] - Base service - Setting up HTTP server
2024-10-21T04:00:37.251048112Z [2024-10-21 04:00:37,250] [    INFO] - Base service - Uvicorn server setup on port 9090
2024-10-21T04:00:37.258530142Z INFO:     Waiting for application startup.
2024-10-21T04:00:37.259042125Z INFO:     Application startup complete.
2024-10-21T04:00:37.259088323Z INFO:     Uvicorn running on http://0.0.0.0:9090 (Press CTRL+C to quit)
2024-10-21T04:00:37.259096032Z [2024-10-21 04:00:37,258] [    INFO] - Base service - HTTP server setup successful
2024-10-21T04:01:03.207863810Z /home/user/comps/agent/langchain/src/strategy/react/planner.py:6: LangChainDeprecationWarning: Importing ChatMessageHistory from langchain.memory is deprecated. Please replace deprecated imports:
2024-10-21T04:01:03.208040048Z
2024-10-21T04:01:03.208054538Z >> from langchain.memory import ChatMessageHistory
2024-10-21T04:01:03.208061913Z
2024-10-21T04:01:03.208067286Z with new imports of:
2024-10-21T04:01:03.208073018Z
2024-10-21T04:01:03.208078195Z >> from langchain_community.chat_message_histories import ChatMessageHistory
2024-10-21T04:01:03.208085274Z You can use the langchain cli to **automatically** upgrade many imports. Please see documentation here <https://python.langchain.com/v0.2/docs/versions/v0_2/>
2024-10-21T04:01:03.208093875Z   from langchain.memory import ChatMessageHistory
2024-10-21T04:01:03.476705209Z /home/user/comps/agent/langchain/src/strategy/base_agent.py:6: LangChainDeprecationWarning: As of langchain-core 0.3.0, LangChain uses pydantic v2 internally. The langchain.pydantic_v1 module was a compatibility shim for pydantic v1, and should no longer be used. Please update the code to import from Pydantic directly.
2024-10-21T04:01:03.476760777Z
2024-10-21T04:01:03.476769082Z For example, replace imports like: `from langchain.pydantic_v1 import BaseModel`
2024-10-21T04:01:03.476775537Z with: `from pydantic import BaseModel`
2024-10-21T04:01:03.476781159Z or the v1 compatibility namespace if you are working in a code base that has not been fully upgraded to pydantic 2 yet.      from pydantic.v1 import BaseModel
2024-10-21T04:01:03.476788679Z
2024-10-21T04:01:03.476794819Z   from ..tools import get_tools_descriptions
2024-10-21T04:10:05.333290359Z The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
2024-10-21T04:10:05.333424504Z INFO:     10.239.182.158:58208 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2024-10-21T04:10:05.346211095Z ERROR:    Exception in ASGI application
2024-10-21T04:10:05.346263775Z Traceback (most recent call last):
2024-10-21T04:10:05.346270588Z   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
2024-10-21T04:10:05.346276128Z     result = await app(  # type: ignore[func-returns-value]
2024-10-21T04:10:05.346280250Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346284535Z   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
2024-10-21T04:10:05.346289968Z     return await self.app(scope, receive, send)
2024-10-21T04:10:05.346294046Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346298000Z   File "/home/user/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2024-10-21T04:10:05.346302406Z     await super().__call__(scope, receive, send)
2024-10-21T04:10:05.346306555Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
2024-10-21T04:10:05.346310970Z     await self.middleware_stack(scope, receive, send)
2024-10-21T04:10:05.346315266Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
2024-10-21T04:10:05.346320215Z     raise exc
2024-10-21T04:10:05.346324275Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
2024-10-21T04:10:05.346328817Z     await self.app(scope, receive, _send)
2024-10-21T04:10:05.346332748Z   File "/home/user/.local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 174, in __call__
2024-10-21T04:10:05.346337226Z     raise exc
2024-10-21T04:10:05.346340935Z   File "/home/user/.local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 172, in __call__
2024-10-21T04:10:05.346345357Z     await self.app(scope, receive, send_wrapper)
2024-10-21T04:10:05.346349190Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
2024-10-21T04:10:05.346353630Z     await self.app(scope, receive, send)
2024-10-21T04:10:05.346357847Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
2024-10-21T04:10:05.346362326Z     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2024-10-21T04:10:05.346366610Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
2024-10-21T04:10:05.346371006Z     raise exc
2024-10-21T04:10:05.346374570Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
2024-10-21T04:10:05.346393075Z     await app(scope, receive, sender)
2024-10-21T04:10:05.346399606Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
2024-10-21T04:10:05.346404820Z     await self.middleware_stack(scope, receive, send)
2024-10-21T04:10:05.346409348Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
2024-10-21T04:10:05.346413670Z     await route.handle(scope, receive, send)
2024-10-21T04:10:05.346417730Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
2024-10-21T04:10:05.346422257Z     await self.app(scope, receive, send)
2024-10-21T04:10:05.346426226Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
2024-10-21T04:10:05.346430560Z     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2024-10-21T04:10:05.346434768Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
2024-10-21T04:10:05.346439237Z     raise exc
2024-10-21T04:10:05.346442897Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
2024-10-21T04:10:05.346447270Z     await app(scope, receive, sender)
2024-10-21T04:10:05.346451117Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
2024-10-21T04:10:05.346455497Z     response = await f(request)
2024-10-21T04:10:05.346459362Z                ^^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346463128Z   File "/home/user/.local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
2024-10-21T04:10:05.346467375Z     raw_response = await run_endpoint_function(
2024-10-21T04:10:05.346471286Z                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346475237Z   File "/home/user/.local/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
2024-10-21T04:10:05.346479615Z     return await dependant.call(**values)
2024-10-21T04:10:05.346483437Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346487790Z   File "/home/user/comps/agent/langchain/agent.py", line 56, in llm_generate
2024-10-21T04:10:05.346492660Z     agent_inst = instantiate_agent(args, args.strategy)
2024-10-21T04:10:05.346497006Z                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346501390Z   File "/home/user/comps/agent/langchain/src/agent.py", line 9, in instantiate_agent
2024-10-21T04:10:05.346506035Z     return ReActAgentwithLangchain(args, with_memory)
2024-10-21T04:10:05.346509928Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346514275Z   File "/home/user/comps/agent/langchain/src/strategy/react/planner.py", line 22, in __init__
2024-10-21T04:10:05.346518568Z     super().__init__(args)
2024-10-21T04:10:05.346526920Z   File "/home/user/comps/agent/langchain/src/strategy/base_agent.py", line 12, in __init__
2024-10-21T04:10:05.346531440Z     self.llm_endpoint = setup_llm(args)
2024-10-21T04:10:05.346536064Z                         ^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346539995Z   File "/home/user/comps/agent/langchain/src/utils.py", line 83, in setup_llm
2024-10-21T04:10:05.346544150Z     model = setup_hf_tgi_client(args)
2024-10-21T04:10:05.346548030Z             ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346551910Z   File "/home/user/comps/agent/langchain/src/utils.py", line 40, in setup_hf_tgi_client
2024-10-21T04:10:05.346556524Z     llm = HuggingFaceEndpoint(
2024-10-21T04:10:05.346560804Z           ^^^^^^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346564875Z   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/load/serializable.py", line 112, in __init__
2024-10-21T04:10:05.346569788Z     super().__init__(*args, **kwargs)
2024-10-21T04:10:05.346573650Z   File "/home/user/.local/lib/python3.11/site-packages/pydantic/main.py", line 212, in __init__
2024-10-21T04:10:05.346577895Z     validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
2024-10-21T04:10:05.346583626Z                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-21T04:10:05.346595464Z pydantic_core._pydantic_core.ValidationError: 1 validation error for HuggingFaceEndpoint
2024-10-21T04:10:05.346602020Z   Value error, Could not authenticate with huggingface_hub. Please check your API token. [type=value_error, input_value={'endpoint_url': 'http://...://10.239.182.158:8080'}, input_type=dict]
2024-10-21T04:10:05.346611397Z     For further information visit https://errors.pydantic.dev/2.9/v/value_error

@xiguiw , I see the error logs said you didn't provide huggingface token:
Value error, Could not authenticate with huggingface_hub. Please check your API token.

Please make sure you have specified below env:
HF_TOKEN="xxxxxxxxxxxxx"
HUGGINGFACEHUB_API_TOKEN=${HF_TOKEN}

@xiguiw , I see the error logs said you didn't provide huggingface token:
Value error, Could not authenticate with huggingface_hub. Please check your API token.

Please make sure you have specified below env: HF_TOKEN="xxxxxxxxxxxxx" HUGGINGFACEHUB_API_TOKEN=${HF_TOKEN}

Yes I found the log messages. The fact is huggingface token is set.
I wonder the "huggingface token" I set it part of the input.

@xiguiw , may you also check the endpoint you provided, I saw logs show something as below
[type=value_error, input_value={'endpoint_url': 'http://...://10.239.182.158:8080'}

in
2024-10-21T04:10:05.346595464Z pydantic_core._pydantic_core.ValidationError: 1 validation error for HuggingFaceEndpoint
2024-10-21T04:10:05.346602020Z   Value error, Could not authenticate with huggingface_hub. Please check your API token. [type=value_error, input_value={'endpoint_url': 'http://...://10.239.182.158:8080'}, input_type=dict]
2024-10-21T04:10:05.346611397Z     For further information visit https://errors.pydantic.dev/2.9/v/value_error

@xuechendi
Thanks Chendi,
it seems agent cannot access local TGI services (the llm endpoint) . After I added ip_address to the no_proxy environment and set no_proxy to the agent. it can access the TGI services.
But it raised another error this time:
Ratelimit.
duckduckgo_search.exceptions.RatelimitException: https://duckduckgo.com/ 202 Ratelimit
2024-10-23T01:34:13.742835233Z > Entering new AgentExecutor chain...
2024-10-23T01:34:13.742842816Z ---
2024-10-23T01:34:13.742848336Z  Knowing the weather helps us plan our day
2024-10-23T01:34:13.742854873Z Action: duckduckgo_search
2024-10-23T01:34:13.742860706Z Action Input: 'weather today in Austin'
2024-10-23T01:34:13.742866428Z ObservationThe token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
2024-10-23T01:34:13.742875048Z Token is valid (permission: read).
2024-10-23T01:34:13.742880653Z Your token has been saved to /home/user/.cache/huggingface/token
2024-10-23T01:34:13.742886173Z Login successful
2024-10-23T01:34:13.742891724Z [DuckDuckGoSearchRun(api_wrapper=DuckDuckGoSearchAPIWrapper(region='wt-wt', safesearch='moderate', time='y', max_results=5, backend='api', source='text'))]
2024-10-23T01:34:13.742897976Z -----------STREAMING-------------
2024-10-23T01:34:13.742903408Z INFO:     10.239.182.158:60956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2024-10-23T01:34:13.760912905Z /home/user/.local/lib/python3.11/site-packages/huggingface_hub/inference/_generated/_async_client.py:2091: FutureWarning: `stop_sequences` is a deprecated argument for `text_generation` task and will be removed in version '0.28.0'. Use `stop` instead.
2024-10-23T01:34:13.761000314Z   warnings.warn(
2024-10-23T01:34:19.292844549Z ERROR:    Exception in ASGI application
2024-10-23T01:34:19.292909574Z Traceback (most recent call last):
2024-10-23T01:34:19.292917586Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 257, in __call__
2024-10-23T01:34:19.292924780Z     await wrap(partial(self.listen_for_disconnect, receive))
2024-10-23T01:34:19.293049607Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 253, in wrap
2024-10-23T01:34:19.293056358Z     await func()
2024-10-23T01:34:19.293061949Z   File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 230, in listen_for_disconnect
2024-10-23T01:34:19.293068815Z     message = await receive()
2024-10-23T01:34:19.293074145Z               ^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293080118Z   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 534, in receive
2024-10-23T01:34:19.293086675Z     await self.message_event.wait()
2024-10-23T01:34:19.293092365Z   File "/usr/local/lib/python3.11/asyncio/locks.py", line 213, in wait
2024-10-23T01:34:19.293098129Z     await fut
2024-10-23T01:34:19.293103345Z asyncio.exceptions.CancelledError: Cancelled by cancel scope 7c96024c43d0
2024-10-23T01:34:19.293109309Z
2024-10-23T01:34:19.293114734Z During handling of the above exception, another exception occurred:
2024-10-23T01:34:19.293120429Z
2024-10-23T01:34:19.293125825Z   + Exception Group Traceback (most recent call last):
2024-10-23T01:34:19.293131369Z   |   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
2024-10-23T01:34:19.293137818Z   |     result = await app(  # type: ignore[func-returns-value]
2024-10-23T01:34:19.293143320Z   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293149115Z   |   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
2024-10-23T01:34:19.293155025Z   |     return await self.app(scope, receive, send)
2024-10-23T01:34:19.293160354Z   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293165735Z   |   File "/home/user/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2024-10-23T01:34:19.293171625Z   |     await super().__call__(scope, receive, send)
2024-10-23T01:34:19.293177015Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
2024-10-23T01:34:19.293182822Z   |     await self.middleware_stack(scope, receive, send)
2024-10-23T01:34:19.293208194Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
2024-10-23T01:34:19.293213247Z   |     raise exc
2024-10-23T01:34:19.293217027Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
2024-10-23T01:34:19.293223629Z   |     await self.app(scope, receive, _send)
2024-10-23T01:34:19.293227785Z   |   File "/home/user/.local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 174, in __call__
2024-10-23T01:34:19.293232389Z   |     raise exc
2024-10-23T01:34:19.293236187Z   |   File "/home/user/.local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 172, in __call__
2024-10-23T01:34:19.293240729Z   |     await self.app(scope, receive, send_wrapper)
2024-10-23T01:34:19.293244665Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
2024-10-23T01:34:19.293249000Z   |     await self.app(scope, receive, send)
2024-10-23T01:34:19.293253020Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
2024-10-23T01:34:19.293257655Z   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2024-10-23T01:34:19.293262034Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
2024-10-23T01:34:19.293267138Z   |     raise exc
2024-10-23T01:34:19.293270780Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
2024-10-23T01:34:19.293275318Z   |     await app(scope, receive, sender)
2024-10-23T01:34:19.293279280Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
2024-10-23T01:34:19.293284329Z   |     await self.middleware_stack(scope, receive, send)
2024-10-23T01:34:19.293288214Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
2024-10-23T01:34:19.293293005Z   |     await route.handle(scope, receive, send)
2024-10-23T01:34:19.293296998Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
2024-10-23T01:34:19.293301338Z   |     await self.app(scope, receive, send)
2024-10-23T01:34:19.293305255Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
2024-10-23T01:34:19.293309540Z   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2024-10-23T01:34:19.293313935Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
2024-10-23T01:34:19.293318334Z   |     raise exc
2024-10-23T01:34:19.293321915Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
2024-10-23T01:34:19.293331895Z   |     await app(scope, receive, sender)
2024-10-23T01:34:19.293336018Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
2024-10-23T01:34:19.293340755Z   |     await response(scope, receive, send)
2024-10-23T01:34:19.293344635Z   |   File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 250, in __call__
2024-10-23T01:34:19.293349038Z   |     async with anyio.create_task_group() as task_group:
2024-10-23T01:34:19.293353009Z   |   File "/home/user/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 685, in __aexit__
2024-10-23T01:34:19.293357307Z   |     raise BaseExceptionGroup(
2024-10-23T01:34:19.293361078Z   | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
2024-10-23T01:34:19.293365205Z   +-+---------------- 1 ----------------
2024-10-23T01:34:19.293369054Z     | Traceback (most recent call last):
2024-10-23T01:34:19.293372827Z     |   File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 253, in wrap
2024-10-23T01:34:19.293377407Z     |     await func()
2024-10-23T01:34:19.293381185Z     |   File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 242, in stream_response
2024-10-23T01:34:19.293385695Z     |     async for chunk in self.body_iterator:
2024-10-23T01:34:19.293389600Z     |   File "/home/user/comps/agent/langchain/src/strategy/react/planner.py", line 59, in stream_generator
2024-10-23T01:34:19.293394292Z     |     async for chunk in self.app.astream(initial_state, config=config):
2024-10-23T01:34:19.293398495Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain/agents/agent.py", line 1810, in astream
2024-10-23T01:34:19.293402938Z     |     async for step in iterator:
2024-10-23T01:34:19.293406834Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain/agents/agent_iterator.py", line 266, in __aiter__
2024-10-23T01:34:19.293411335Z     |     async for chunk in self.agent_executor._aiter_next_step(
2024-10-23T01:34:19.293415440Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain/agents/agent.py", line 1556, in _aiter_next_step
2024-10-23T01:34:19.293419847Z     |     result = await asyncio.gather(
2024-10-23T01:34:19.293423638Z     |              ^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293427425Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain/agents/agent.py", line 1589, in _aperform_agent_action
2024-10-23T01:34:19.293431945Z     |     observation = await tool.arun(
2024-10-23T01:34:19.293435820Z     |                   ^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293439807Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/tools/base.py", line 808, in arun
2024-10-23T01:34:19.293444185Z     |     raise error_to_raise
2024-10-23T01:34:19.293448095Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/tools/base.py", line 775, in arun
2024-10-23T01:34:19.293457149Z     |     response = await asyncio.create_task(coro, context=context)  # type: ignore
2024-10-23T01:34:19.293462938Z     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293467000Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/tools/base.py", line 575, in _arun
2024-10-23T01:34:19.293471658Z     |     return await run_in_executor(None, self._run, *args, **kwargs)
2024-10-23T01:34:19.293475652Z     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293479627Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 590, in run_in_executor
2024-10-23T01:34:19.293484098Z     |     return await asyncio.get_running_loop().run_in_executor(
2024-10-23T01:34:19.293488127Z     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293492095Z     |   File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
2024-10-23T01:34:19.293496458Z     |     result = self.fn(*self.args, **self.kwargs)
2024-10-23T01:34:19.293500625Z     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293504647Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 581, in wrapper
2024-10-23T01:34:19.293509081Z     |     return func(*args, **kwargs)
2024-10-23T01:34:19.293512987Z     |            ^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293516940Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain_community/tools/ddg_search/tool.py", line 73, in _run
2024-10-23T01:34:19.293521349Z     |     return self.api_wrapper.run(query)
2024-10-23T01:34:19.293525125Z     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293529015Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain_community/utilities/duckduckgo_search.py", line 97, in run
2024-10-23T01:34:19.293533674Z     |     results = self._ddgs_text(query)
2024-10-23T01:34:19.293537645Z     |               ^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293541618Z     |   File "/home/user/.local/lib/python3.11/site-packages/langchain_community/utilities/duckduckgo_search.py", line 64, in _ddgs_text
2024-10-23T01:34:19.293546918Z     |     ddgs_gen = ddgs.text(
2024-10-23T01:34:19.293551265Z     |                ^^^^^^^^^^
2024-10-23T01:34:19.293555065Z     |   File "/home/user/.local/lib/python3.11/site-packages/duckduckgo_search/duckduckgo_search.py", line 239, in text
2024-10-23T01:34:19.293559465Z     |     results = self._text_api(keywords, region, safesearch, timelimit, max_results)
2024-10-23T01:34:19.293563569Z     |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293567909Z     |   File "/home/user/.local/lib/python3.11/site-packages/duckduckgo_search/duckduckgo_search.py", line 273, in _text_api
2024-10-23T01:34:19.293577535Z     |     vqd = self._get_vqd(keywords)
2024-10-23T01:34:19.293581575Z     |           ^^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293585360Z     |   File "/home/user/.local/lib/python3.11/site-packages/duckduckgo_search/duckduckgo_search.py", line 134, in _get_vqd
2024-10-23T01:34:19.293590235Z     |     resp_content = self._get_url("POST", "https://duckduckgo.com", data={"q": keywords})
2024-10-23T01:34:19.293595707Z     |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-23T01:34:19.293599925Z     |   File "/home/user/.local/lib/python3.11/site-packages/duckduckgo_search/duckduckgo_search.py", line 129, in _get_url
2024-10-23T01:34:19.293604360Z     |     raise RatelimitException(f"{resp.url} {resp.status_code} Ratelimit")
2024-10-23T01:34:19.293608938Z     | duckduckgo_search.exceptions.RatelimitException: https://duckduckgo.com/ 202 Ratelimit
2024-10-23T01:34:19.293613247Z     +------------------------------------

The

will replace duckduckgo_search with https://github.com/opea-project/GenAIExamples/tree/main/SearchQnA microservice and give a simple example for custom tool

authors: lkk12014402 xiguiw xuechendi

repo: opea-project/GenAIComps


text:
Separate API to *_types.go files

multiple apis are put in one single types.go file which make it hard to read. It is better that we can separate them to multiple files.

/assign @ilan-pinto

repo: open-cluster-management-io/api


text:
Add KEP for AddOnPlacementScoreGenerator

As discussed at the community meeting on April 6th, this is the KEP for adding the AddOnPlacementScoreGenerator API.

I am still not convinced that this should be put in the API repo. The major uncertainty is how this API can be evolved and consumed by multiple score providers. It seems to me a new score provider has to update this API or build their own CRDs to config how score is generated. It does not seems change the fact that AddOnPlacementScore is the only integration point. But I am 100% ok if we create a separated repo and maintain this API and controller there.

I'm not sure I follow you. The idea behind this CRD is to allow the customization of score generations by introducing a new API that can be consumed by the addon controllers, I don't think we want to have a dedicated controller for this API.
Multiple score providers will be required to read this API in case they want to get score-generation customizations. In case these score providers don't need customizations they're fine not reading this CRD.
To me, this API would be like having your config in a ConfigMap and reading it upon controller start.

I think I understand the intention, but I do not see how it can be achieved when a new provider comes without adding additional field in this API. For example, if I have a new provider comes and allow user to configure it to score the cluster based on resource usage, I have to update the API spec or introduce a new API to config. I do not see how you can define a common api schema that can adapt to different providers.

I think I understand the intention, but I do not see how it can be achieved when a new provider comes without adding additional field in this API. For example, if I have a new provider comes and allow user to configure it to score the cluster based on resource usage, I have to update the API spec or introduce a new API to config. I do not see how you can define a common api schema that can adapt to different providers.

That's why I think it should be unstructured, so anyone can put their customization data without having to change the API. The controllers are responsible for reading their data and making sure that in case of misconfiguration, they can gracefully handle it.

That's why I think it should be unstructured, so anyone can put their customization data without having to change the API. The controllers are responsible for reading their data and making sure that in case of misconfiguration, they can gracefully handle it.
if it is unstructured, why we need to define an API schema instead of directly using a configmap :)? I think we need to at least  define some general concept in this API rather than arbitrary type...This is why I am struggling with how this API can provide commonality in this scenario.

if it is unstructured, why we need to define an API schema instead of directly using a configmap :)? I think we need to at least define some general concept in this API rather than arbitrary type...This is why I am struggling with how this API can provide commonality in this scenario.

It could be done via ConfigMaps, we won't be able to leverage things defined in the KEP like addOnPlacementSelector  but I guess that's not that bad.
I cannot think of a common structured API that can be leveraged without having to update API types for every new provider. Requirements for different data providers may be very different and as such, hard to put in common.

repo: open-cluster-management-io/enhancements


text:
enable github actions for presubmit and postsubmit

Signed-off-by: Mike Ng ming@redhat.com

/lgtm


text:
Add unit test for PlacementRule

Signed-off-by: Maggie Chen magchen@redhat.com

/hold

/lgtm


text:
use a more appropriate name for github action that publish the built image

With my limited knowledge of GitHub actions, I am assuming this value can be changed to a more fitting name.
Signed-off-by: Mike Ng ming@redhat.com

/approve
/lgtm


text:
add imageRegistry role

Signed-off-by: Zhiwei Yin zyin@redhat.com

/lgtm


text:
Undeploy script

This undeploy script (AKA remove.sh) will look for matching <prefix>-chan Channel and <prefix>-sub Subscription resources and remove them in tandem.
Neither a namespace nor a prefix is required--the script will poll the cluster for both if not provided.
If the Subscription has been changed and no longer points to the Channel, the script refuses to delete the resource but gives the user a warning and the commands to do it themselves.

adding @dockerymick to take a look at doc

Is oc a hard requirement for user to run this script? Can we do the same with just kubectl?

Yes, oc is a hard requirement because it allows the script to find and iterate over the available namespaces. I'm not aware of a way to find available namespaces otherwise.
I changed deploy.sh to use oc also, but I can change it back to kubectl if it needs to be more generic. I was thinking the scripts would both be pointing at an Openshift RHACM hub, but maybe that's a poor assumption? If so, I can adjust the script to require a namespace be provided if a user doesn't have clusterwide access.

I think it's not a problem for now. But in the future, after we open source the framework code, it should not require openshift to deploy and test. So it's better to make oc optional.

Ah, okay. Let me make that change now so that we don't have to worry about it later.

@dockerymick I made some additional adjustments to your edits. Most notably, I replaced "View the following example..." with "View the following guidance..." since to me it's not an example. And I also added a note that parameters are optional. What do you think?

@dhaiducek great idea, i agree with your change

authors: dhaiducek dockerymick ycao56


text:
Sentry Integration

Integrate Sentry to the backend project
https://docs.sentry.io/platforms/python/guides/fastapi/

I've sent the code via Slack. Noting that we'll also want Sentry for the frontend.

@nahu Link to the PR
https://github.com/open-contracting/credere-backend/pull/51

authors: adrianosvaldomartinez jpmckinney nahu

repo: open-contracting/credere-backend


text:
Proxy /api/cases calls manually in curator frontend

The reason I'm going back to this is because once we add auth (which is the thing I'm doing now), we'll authenticate the user in the curator service and change the post request to contain its id (for storing contribution information), amongst other kind of data-mangling thing we're going to do in the curator service before sending the request to the data service.
I went all-in and removed the http proxy library I added yesterday to keep things consistent, I didn't want to proxy only certain methods and not others to keep things simple and in one place.

 Set up a login and auth mechanism.

repo: open-covid-data/healthmap-gdo-temp


text:
Removing domain?

Greetings :)
Hope you are doing well.
How do I remove a registered domain on Open-domains?
I registered a domain, it was approved, but I only realised afterwards that I had the spelling incorrect.
Thank you,
Have a good day ahead.

You can delete the file and open a new pull request with the new one.

Thank you, I appreciate the guidance :)

authors: DEV-DIBSTER SpaceIgniter

repo: open-domains/register


text:
Create ccds.is-cool.dev

Requirements

[x] You have completed your website.
[x] The website is reachable.
[x] The CNAME record doesn't contain https:// or /.
[x] There is sufficient information at the owner field.
[x] Your website is not hosting on the following due to SSL issues: Vercel, Netlify

Description
It will be used for personal portfolio to showcase thing about myself.
Link to Website
http://103.81.70.71

CLA Assistant Lite bot: Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

I have read the CLA Document and I hereby sign the CLA

You can retrigger this bot by commenting recheck in this Pull Request

I have read the CLA Document and I hereby sign the CLA

authors: OD-BOT rafathasan


text:
Add key pass domain

Requirements

[x] You have completed your website.
[x] The website is reachable.
[x] The CNAME record doesn't contain https:// or /.
[x] There is sufficient information at the owner field.
[x] There is no NS Records (Enforced as of Sepetember 4th, 2024)

Description

This domain will be used for a access cards backend system
Link to Website

https://access-cards-tudg.onrender.com/

I have read the CLA Document and I hereby sign the CLA


text:
graph模块经常出现OOM

两个graph同一套代码，一个正常，另一个经常OOM，不知道是不是hash的不均匀？

运行时free看看，内存占用情况和内容总大小。
hash不均理论上是有可能的

@imzye 有进一步信息吗

将两个大内存的graph换成4个稍小内存的graph，修改transfer配置后就没再出现这个问题。@laiwei
之前我尝试修改graph源码里面的DefaultMaxCacheSize，CLEAN_CACHE，FLUSH_DISK_STEP，FLUSH_MAX_WAIT这些常量参数，重编译运行一段时间后（3 - 9小时）都会出现上述OOM问题，不知道是否是因为磁盘的随机写性能太差，导致数据来不及落到盘上，最终OOM。

authors: imzye laiwei

repo: open-falcon/falcon-plus


text:
Do parm@frosst energy comparison to validate SMIRFF implementation

We need to process the parm@frosst molecules beginning from both AMBER (.prmtop and .crd) files and SMIRFF .xml format with OEMol objects to ensure that we get the same energies (within reasonable tolerance) for the same system implemented both ways, as validation for the .xml files implemented in #24 .
Ultimately some sort of check along these lines should also be implemented in the tests.
I am working on this.

Partly resolved by #38 , but more testing remains to be done once #41 is resolved.

Closed by #60 .

repo: open-forcefield-group/smarty


text:
Alerts - no alert raised when a User is created or deleted

Affected module
UI (and possibly backend)
Describe the bug
Alerts - no alert raised when a User is created or deleted.
To Reproduce
Add an alert within OM's UI for "User Created or Deleted":

Trigger: User
Filter: Event Type - Entity Created, Entity Deleted, Entity Soft Deleted - Include
Add a valid Slack webhook or email integration with "Send to: Admins" ticked.

Expected behavior
When a new user is created in OM, then an alert should be received via Slack or email to say that the user has been created.
Version:

OS: macOS Ventura 13.6
Python version: 3.9.18
OpenMetadata version: 1.1.5
OpenMetadata Ingestion package version: 1.1.5

Additional context
The issue is that no alert is being received when a user is created or deleted within OM.

To Reproduce
Configure an alert as follows:
Trigger: tick "All".
Filter: Event Type - Entity Created, Entity Updated, Include
Add a valid Slack webhook or email integration with "Send to: Admins" ticked.
Change the owner of an entity within OM to a user within OM.
Delete the user who you just changed the entity to be owned by.
Expected behavior
An alert should be received stating that the entity's owner has been changed from the deleted user to "No Owner".
To Reproduce
Configure an alert as follows:
Trigger: untick "All", tick all of the sub-options within "All".
Filter: Event Type - Entity Created, Entity Updated, Include
Add a valid Slack webhook or email integration with "Send to: Admins" ticked.
Create a new Container or modify an existing Container within a storage service in OM.
Expected behavior
"Container" should appear in the list of entities within the "Trigger" section of the alert (if "All" is deselected).
It should be possible to select "Container" as the only entity type within the trigger, and "Event Type === entityCreated, entityUpdated" for the filter, together with a valid Slack or email integration.
When a new Container is created, or an existing Container is updated, an alert should be received via Slack or email

authors: harshach ruth-mills-naimuri

repo: open-metadata/OpenMetadata


text:
Automatically publish Docker Images

Prepare a github action capable of building, tagging and publishing all new docker images after a release or when a bugfix/patch is required on demand

@akash-jain-10 @Vj-L ^

This is partially done for release! we need to update CI to make is run on BugFix and Patches!

@akash-jain-10 we might just need to add something like
on:
  workflow_dispatch:
    inputs:
      tag:
        description: "Release tag being built"

so that we can pass the tag or whatever info we need to send and allow the manual trigger

authors: akash-jain-10 harshach pmbrull


text:
TestCase results are not shown if the default lookup for last 3 days doesn't have any data

Affected module
UI
Describe the bug
If there is no data available in last 3 days we are not showing a selection of duration all together.

To Reproduce
Screenshots or steps to reproduce
Expected behavior
A clear and concise description of what you expected to happen.
Version:

OS: [e.g. iOS]
Python version:
OpenMetadata version: [e.g. 0.8]
OpenMetadata Ingestion package version: [e.g. openmetadata-ingestion[docker]==XYZ]

Additional context
Add any other context about the problem here.

@ShaileshParmar11 No data description should more descriptive here.

authors: chirag-madlani harshach


text:
Fixes #11883 - A user can resolve a task created by him

Describe your changes:
Fixes #11883

I worked on ... because ...

Type of change:

[x] Bug fix
[ ] Improvement
[ ] New feature
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] My PR title is Fixes <issue-number>: <short explanation>
[ ] I have commented on my code, particularly in hard-to-understand areas.
[ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

Test failures seem unrelated to this patch. Merging the PR.


text:
FIXES #14772: Display Assets without Owner/Description

Describe your changes:
Fixes #14772
In order to display assets without Owner/Description we have the following approach:
From the Data Insights page a user will be able, by the click of a button, to land on the Explore page with Advance Filters already applied (Either to get the assets with incomplete description or without owner).
This filters will be persistent across Asset Types and they will be able to navigate freely.
In order to achieve that:

We already have the filter available for the Owner attribute
For description we are creating a new pre-computed attribute descriptionStatus that will either resolve to 'INCOMPLETE' or 'COMPLETE'.
A new Advanced Filter will be added for this attribute
UI modified accordingly

Type of change:

[x] Improvement

Checklist:

[x] I have read the CONTRIBUTING document.
[x] My PR title is Fixes <issue-number>: <short explanation>
[ ] I have commented on my code, particularly in hard-to-understand areas.
[ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

💡 Maven MySQL failure related to ES test flakiness


text:
HELP! low accuracy on once dataset using pointrcnn.

I trained pointrcnn on once dataset for 80 epochs with the default config.
But got very poor accuracy as follows:

I wonder where's the problem, did I set the config wrong?
could u please give me some advice on how to improve the performance?
Many thanks!

The scores are as expected and similar to what was reported in the paper.
To further improve, you may try changing to a smaller batch size or adjusting the ball query radius, which might help a little.

Hi, thanks for your reply, why smaller batch size would help? do you mean set a bigger batch size?

I didn't try but I heard using batch size 1 can boost the AP by 2%
https://github.com/PointsCoder/ONCE_Benchmark/issues/31#issuecomment-1311804549

I guess it is because the training set is small so different batch sizes bring some randomness on the performances.

authors: DurbinLiu PointsCoder

repo: open-mmlab/OpenPCDet