text:
RFC: blueprint acquisition system

Right now it's kind of annoying to use third party blueprints in stacker - you have to get them (git pull), then install them in your python path (python setup.py install), only then can you use them.
I'm trying to think of a better way to do this, so opening this up for discussion from 'the community' :)

One idea is to do something like golang's import system:
github.com/phobologic/myblueprints/myblueprints/cool_blueprint.CoolBlueprint

Then make it so that stacker pulls that file down (ending with .py, the CoolBlueprint part is the class), puts it in a temp directory, then adds that temp directory to your python path while it tries to import the blueprints.

i vote for:

current working directory gets appended to sys.path so you can access local blueprints
i agree with something like the above, although i would maybe push for something similar to a Gemfile:

# conf/example.yaml

stacks:
  - name: coolBluePrint
    blueprint:
      git: github.com/phobologic/myblueprints
      path: myblueprints.cool_blueprint.CoolBlueprint

i think that gives you more control than just having something like "blueprint: github.com/phobologic/myblueprints/myblueprints/cool_blueprint.CoolBlueprint" or "blueprint: stacker.blueprints.vpc.VPC", and having the logic be driven by the prefix of blueprint.
i would rather have something more verbose that gives you more room to expand it in the future.
i would also want to account for something like we did for empire stacks where the blueprint used other python files, ie "policies.py", which means more than just downloading 1 file.

Either of those would work.  Another alternative is to set a list of directories where Stacker would look for blueprints...set in the yaml file:
blueprint_dirs:
    - ~/path/to/blueprints
    - ../another/path/to/blueprints

That coupled with the ideas above could work well...i'd also suggest changing some names...source and class to be more general and specific, respectively:
stacks:
  - name: coolBluePrint
    blueprint:
      # Only look in this repo for this blueprint
      source: github.com/phobologic/myblueprints
      class: myblueprints.cool_blueprint.CoolBlueprint
  - name: cache
    blueprint:
      # this would look through the set of blueprint_dirs, in addition to any default dirs
      class: sekrt.elasticache.ElasticCache

+1 to what @brianz is suggesting. i think that could work really well

Generally +1 although fetching is a slippery slope given "go get" supports multiple VCS and then dealing with proxies, etc. Also, adding a way to specify branch or hash would be good for testing. See? Slippery slope of features...

What @markpeek said...totally agree.  As a first step it'd be nice to simply add the ability to import local files and leave the other stuff as an exercise to the developer.
Yet another option....create your own blueprint packages, pip install them, done.

the local paths are the most important to me, i agree with the pip install one too

diff --git a/stacker/util.py b/stacker/util.py
index 3f714a1..548ca9a 100644
--- a/stacker/util.py
+++ b/stacker/util.py
@@ -1,5 +1,6 @@
 import importlib
 import logging
+import os
 import re
 import sys
 import time
@@ -83,7 +84,11 @@ def load_object_from_string(fqcn):
     object_name = fqcn
     if '.' in fqcn:
         module_path, object_name = fqcn.rsplit('.', 1)
-        importlib.import_module(module_path)
+        try:
+            importlib.import_module(module_path)
+        except ImportError:
+            sys.path.append(os.path.curdir)
+            importlib.import_module(module_path)
     return getattr(sys.modules[module_path], object_name)

the version of stacker I run just has the above change and I haven't had to think much about it after that. I think blueprint_dirs would be a better option though.

So, I think the blueprint_dirs idea works for most cases.  Our workflow is that we have a repo private_stacks that has a setup.py.  We have to make sure that before we update an environment we pull down the latest private_stacks and run python setup.py install in the virtualenv we're using
The blueprint_dirs suggestion wouldn't help with ensuring the git repo is up to date, but it would help with not having to do python setup.py install, though it also means that everyone has to have the same layout of git directories.  Hmmmm.  Not sure how to solve that problem.

what if you added a pre_hook to pull the repo?

So....I'm liking the idea of blueprints as packages a lot.  The reason are:

zero changes needed to stacker
you can pip install -e . your package, and always get up-to-date blueprints while developing
you can alternately treat the blueprints as versioned requirements for your system, just like any other python package
IMO, updating a versioned package is outside the scope of this tool and should be left up to the user
other people could pip install your blueprints, which would mean starting to build a library of shared components like we're all used do: docker hub, chef recipes, etc.
this is a very low barrier to entry for people...provide a single example of an installable blueprint, and people would be able to figure it out really quickly.  You could even supply a cookie cutter recipe to create a blueprint: https://github.com/audreyr/cookiecutter

Something as basic as this works, again, with zero changes to stacker.
from setuptools import setup, find_packages

setup(
    name='mycompany_blueprints',
    version='0.0.1',
    author='Brian Zambrano',
    author_email='brianz@gmail.com',
    url="https://github.com/brianz/mycompany-blueprints",
    description='Stacker blueprints for My Company',
    packages=find_packages(),
)

i buy all of that +1

I think the introduction of stacker_blueprints as a separate package that has dependencies on specific versions of stacker mostly handles this.  For personal packages, the same process should be followed (and if your stacks rely on stacker_blueprints, it should specify that in setup.py, required_packages with a specific version).  Closing this out.

authors: brianz markpeek mhahn phobologic


text:
Swift 2 compliant

Gents:  Good job on this little frame-work.  As you know, Swift 2 discontinued "reflect" and introduced "Mirror" class instead.  I've changed your wonderful code and made it Swift-2 compliant.  If you want, I can submit the code here.
Kind regards.

Great, you can make a pull request. Thank you, could be a good start for update this framework.

source: gharchive/issue

repo: remirobert/Camembert

license: mit

license_type: permissive

license_source: bigquery


text:
Add overrides applier

Hi!
I noticed one of the known issues about overrides and tried to create a solution to this problem.
I'm not sure if my method of copying props is correct or not. However, I didn't find anything in docs about copying styles and made it this way :–)
The function is async now. I had to convert it, because in order to copy font styles we need to load fonts used by nodes. Figma API has a method called figma.loadFontAsync which loads required font, but it is async.
If there is something else I can help with, I would be glad to :–)

Hi @bespoyasov ! Thanks for the contribution, this would be amazing to be as a configuration option... I often need a clean copy of the component!
I just need to merge @mikeozornin PR and them try to solve the conflicts for this one, okay?

@bespoyasov, I tried to help in testing of your version, cloned your repo and created a local plugin.
Get this error, and plugin still running (maybe async not closed):

Screencast: http://d.mikeozornin.ru/4KrGPN
Test file:
https://www.figma.com/file/tM2Dw30LOarrP7GbDEUK8h/reattach-test-file?node-id=0%3A1

Hi @renancamm,
I was wandering if there is anything else I can help with?

@bespoyasov I have files where overrides sometimes are lost.
Here is screencast: http://d.mikeozornin.ru/GUsK4M
There are two frames.
Lower one is created by detaching the component instance, reattach works like a charm.
Upper one is taken from some my file mockup and detaching doesn't work. Frames structure seems identical (close icon differ, but its overrides could by lost).
It's not a critical, but maybe is't some interesting case for testing.
Here is file, everybody has edit permissions: https://www.figma.com/file/Z1Eq4qFVJYc3bKNFOcp5Ca/Reattach-test-file?node-id=0%3A1

@mikeozornin, I reproduced the issue, but seems like I accidentally reloaded the page while debugging and lost that broken component :–(
How can I recreate the env to get the issue again?

@bespoyasov Done. I restored version before your changes.
There are several frame copies, just make one more copy before experimenting.
The same link: https://www.figma.com/file/Z1Eq4qFVJYc3bKNFOcp5Ca/Reattach-test-file?node-id=0%3A1

@mikeozornin thanks a lot!
I copy a file to local files since then :–)
Anyway, I've investigated this issue and it seems like it had something to do with the structure of a frame. On a screenshot below I tried to show the difference between the subtrees, which lead to an error:

The children count didn't match, so the plugin couldn't apply overrides to a node, which didn't exist.
Since we cannot predict which structures are going to be there and which one is correct I ignore frames with different structures. The plugin is still able to apply overrides, but if it encounters the difference it won't apply them.
Different structures here are just a different count of children in frames. I don't compare each subtree node with that in a different frame/instance, because they may have different names.
Now it is working as expected, however it might need further investigation for more complex cases.
Also, I fixed the error rendering on the main screen of the Figma App. Now it tells what happened right in the bottom message panel:

Somehow I forgot about it when refactored last time and all the errors passed silently in the console, which wasn't the goal really. Sorry about that.

Expected behavior for me: save overrides for frames parts with same structure and ignore parts that have another name or structure. Also I expect reseting overrides to whole subtree of node, that doesn't match by name or structure.
Now plugin works like I expect. Great thanks!

Tested with team lib.
Everything works :–)

Okay guys, I'm back
I was moving to another continent... A lot of trouble in the last weeks 😱
Let's merge and push to figma 💪

I was moving to another continent... A lot of trouble in the last weeks 😱

Wow :–)

@bespoyasov thanks for the amazing contribution!
The new code is working most of the time, but sometimes if it's freezing when I select frames that have something to copy from and other that don't have any reference.
I can't replicate this behaviour on new files tho. Do you know about any known issue?

@renancamm that's interesting, I haven't seen this yet, but I want to investigate it... 🤔
I'm going to need some more info. Could you help me, please?

How is it freezing?
Does the plugin finish the process or just crashes with an error?
Do those frames have something inside of them? If so, what is it: a text content, another frame, a component etc?

If there is some file with a problem components/frames it would be a huge help.

How is it freezing?

It get stuck in this:

Does the plugin finish the process or just crashes with an error?

No crash, no error. Console is clean! My bet is on some async problem.

Do those frames have something inside of them? If so, what is it: a text content, another frame, a component etc?

A bit of everything
Here is a sample file: https://www.figma.com/file/7YgwLVPcGC4USsXFoAntHA/Reattache-Plugin-Debug-sample?node-id=0%3A1

@renancamm I looked at it. Figma tells that there is an issue with nested component:
Can't create instance of nested master component. Please call figma.importComponentByKeyAsync(symbol.key) and await the result. The resulting component can be using createInstance().

I tried to call this figma.importComponentByKeyAsync() method, but it tells that it cannot find a component with given key among the published components.
I'm not sure if this is the issue with the code itself or with the components library. It seems like I'm going to need some more info:

Is the library included in your case?
Are the components "published" (what does it mean, btw)?
How can I know if components have different "keys"?

We can use some try-catch workarounds to avoid this issue, but I wouldn't go with this idea until we know what exactly what is going on.

@bespoyasov
Is the library included in your case?
The components are linked to a team library
Are the components "published" (what does it mean, btw)?
yes, otherwise it would't be possible to see them in other files.
("publish" is how you make changes in team libraries visible to other files)
How can I know if components have different "keys"?
No idea

So, I did a bunch of tests and it seems that the real problem is components from team libraries. It was working before, so I guess I lost track when it stopped working.
Do you know how to debug this beside printing everything to console?
I'm going to pay more attention to what is happing is this line:
let instanceClone = instanceReference.masterComponent.createInstance();

Do you know how to debug this beside printing everything to console?
¯_(ツ)_/¯
Only debugger — it helps to “stop” the program at a given moment and inspect the context.

@bespoyasov Updates from my side: It always fail with team library components (even in files inside my team) when we get to this line: let instanceClone = instanceReference.masterComponent.createInstance();, but it's failing silent because of async function.
I tested with the previous version, with #6 version, and the initial code version. None of them seem to work... It's weird because I could swear that the initial code worked in this case, because that's my only use case. They could have changed the API access to team library or I got confused somehow. I will investigate this further.

@bespoyasov @mikeozornin  While testing the code, I got surprised by #6 code. What do you think if we change to that codebase?

source: gharchive/pull-request

authors: bespoyasov mikeozornin renancamm

repo: renancamm/figma-reattache-instance

license: MIT

license_source: github-api


text:
How to make renative use my forked version of react-native?

I've been developing an androidtv application using renative and it's going well but unfortunately, we encountered some random crushes. Long story short, it is an issue that happens on some android 12 devices and is still not fixed by react-native team but can be somewhat mitigated by modifying react-native java code. I forked the react-native-tvos plugin, modified the code, and installed it as a custom node module through package.json. Unfortunately, I can't figure out how to make renative use this version of react-native-tvos instead of the original one. How do I configure renative to do so?

@maciejsiiminski thanks for this question, we will be looking into this

authors: maciejsiiminski pauliusguzas

repo: renative-org/renative


text:
Children not exiting with overviewer

Children are not killed when overviewer quits. This is problematic with long running scripts such as servers etc.

A simple process.on('exit', () => child.kill()) can do the trick. However I notice that you use webpack to compile your published files and it will shim process and it's not a good thing. You should probably compile your lib using babel alone. Is there a reason for you to use webpack here?

Hi @Yomguithereal
I added your solution in e68764205bdb37e0eccb87e8af0e87c955f43396 and published a new minor version – I am pretty sure what webpack does not shim process because I configured to build a bundle for Node (and not the browser) so it should work.

@renke I am still having an issue with child processes not exiting. Any advice?
I tried a variety of similar packages and this issue happened with all of the following:

http-server
lr-http-server
webpack-dev-server

The first time, everything runs fine. If I quit overviewer and start it again, I get "port is already in use" type warnings. I then have to manually kill the process before restarting overviewer.

authors: Yomguithereal afaqurk renke

repo: renke/overviewer

license: isc


text:
How to import this library through cdn in Vue

Vue can not recongnize gifsicle from cdn,maybe something wrong

npm-install

authors: butterPig renzhezhilu

repo: renzhezhilu/gifsicle-wasm-browser


text:
界面

如何自己定义播放器的界面

我有提供自定义的代理

你的demo里没有这个代理呀，能说仔细一点吗

repo: renzifeng/ZFPlayer


text:
关于[ZFPlayerControlView zf_playerHideControlView]崩溃的问题

通过bugly日志分析，发现这个崩溃很频繁，bugly上解释说是引用了一个无效的内存，具体的崩溃堆栈如下图：

这个是什么原因呢？实在是反馈日志上太频繁了，我又查不出来原因，请大神指点

这里代理被设为 assign 的关联对象,有可能被提前释放掉
ZFPlayer/ZFPlayer/Category/UIView+CustomControlView.m
- (void)setDelegate:(id<ZFPlayerControlViewDelagate>)delegate {
    objc_setAssociatedObject(self, @selector(delegate), delegate, OBJC_ASSOCIATION_ASSIGN);
}

这样的话,在下面这段代码调用
[self.delegate respondsToSelector:@selector(zf_controlViewWillHidden:isFullscreen:)]
会向一个已经释放的对象发送消息
ZFPlayer/ZFPlayer/ZFPlayerControlView.m
- (void)zf_playerHideControlView {
    if ([self.delegate respondsToSelector:@selector(zf_controlViewWillHidden:isFullscreen:)]) {
        [self.delegate zf_controlViewWillHidden:self isFullscreen:self.isFullScreen];
    }
    [self zf_playerCancelAutoFadeOutControlView];
    [UIView animateWithDuration:ZFPlayerControlBarAutoFadeOutTimeInterval animations:^{
        [self hideControlView];
    } completion:^(BOOL finished) {
        self.showing = NO;
    }];
}

解决方法见
https://stackoverflow.com/a/27035233/6777996

@guyuliunian 请问问题解决了吗？

好的，谢谢

我也遇到了这个问题，请问你最后是怎么解决的呢？

authors: Minozmin TaoistKing WeilBJDE XeonPhil guyuliunian


text:
暴漏AVPlayerLayer，为提供给画中画使用，增加简单的画中画开启样例

在ZFAVPlayerManager中暴漏AVPlayerLayer，
为提供给画中画使用，增加简单的画中画开启样例

+1

在ZFAVPlayerManager中暴漏AVPlayerLayer，
为提供给画中画使用，增加简单的画中画开启样例

画中画直接自定义按钮 跳转AVPictureInPictureController

+1

在ZFAVPlayerManager中暴漏AVPlayerLayer，
为提供给画中画使用，增加简单的画中画开启样例

我现在这样处理的，自己建立一个avplayer然后加载和zfplayer一样的url，实现了画中画效果

已经在.h文件添加了AVPlayerLayer，可以使用画中画，但是正在播放视频时，APP进入后台后画中画会自动弹出，demo地址：https://github.com/howhyone/FDPictureInPicture，有兴趣的可以看下是什么情况

已经在.h文件添加了AVPlayerLayer，可以使用画中画，但是正在播放视频时，APP进入后台后画中画会自动弹出，有兴趣的大佬看下是什么情况，demo地址：https://github.com/howhyone/FDPictureInPicture

self.pipController = [[AVPictureInPictureController alloc] initWithPlayerLayer:playerManager.playerLayer];
self.pipController.delegate = self;

先不要写

已经在.h文件添加了AVPlayerLayer，可以使用画中画，但是正在播放视频时，APP进入后台后画中画会自动弹出，有兴趣的大佬看下是什么情况，demo地址：https://github.com/howhyone/FDPictureInPicture

self.pipController = [[AVPictureInPictureController alloc] initWithPlayerLayer:playerManager.playerLayer];
self.pipController.delegate = self;

先不要写

不写就没在APP内也没画中画效果了，需求是要开启画中画的

已经在.h文件添加了AVPlayerLayer，可以使用画中画，但是正在播放视频时，APP进入后台后画中画会自动弹出，有兴趣的大佬看下是什么情况，demo地址：https://github.com/howhyone/FDPictureInPicture

self.pipController = [[AVPictureInPictureController alloc] initWithPlayerLayer:playerManager.playerLayer];
self.pipController.delegate = self;

先不要写

不写就没在APP内也没画中画效果了，需求是要开启画中画的

你要开启的时候再写，
记得加个延迟
startPictureInPicture

已经在.h文件添加了AVPlayerLayer，可以使用画中画，但是正在播放视频时，APP进入后台后画中画会自动弹出，有兴趣的大佬看下是什么情况，demo地址：https://github.com/howhyone/FDPictureInPicture

self.pipController = [[AVPictureInPictureController alloc] initWithPlayerLayer:playerManager.playerLayer];
self.pipController.delegate = self;

先不要写

不写就没在APP内也没画中画效果了，需求是要开启画中画的

你要开启的时候再写，
记得加个延迟
startPictureInPicture

大佬牛批，这样写就可以了，当不用AFPlayerLayer库时， 原来的写法是没问题的，不知道是有什么骚操作

authors: PZXforXcode howhyone xxxIxxxx


text:
RUN-1522: fix and guard asserts related to script invocations

Explanation: https://linear.app/replay/issue/RUN-1488#comment-8f054abf
-> This also includes https://github.com/replayio/chromium-v8/pull/112.

Add some asserts in chromium-v8
Mark our own code execution paths with EventsDisallowed
Test here: 95 passed, 5 failed

Crashes here: 38 Pause.evaluateInFrame crashes ❌

Re-tested here: 45 passed, 4 failed (50 total)

Crashes here: Looks fine. ✔

@bhackett1024 After adding these guards, I get a bunch of these command crashes (see here). It appears that disallowing events during SetupRecordReplayCommands (or maybe in SendMessageToFrontend) made things take different paths in such a way that the scope is now affected by it?
What's even weirder is that I do not see this locally after running a bunch of tests. Might be hard to pin down.
Open to any suggestions.
{
  "Chromium": true,
  "Download Speed (Mbps)": 0,
  "Gecko": false,
  "Recordings Prefix": false,
  "Recordings S3 Bucket Prefix": false,
  "Resources Prefix": false,
  "Timestamp": "2023-03-21T11:01:29.198Z",
  "coalesced.buildId": "linux-chromium-20230321-1cd1ac564ffe-bfa354b141e5",
  "controlId": "b9ab764b-5b04-4c56-9f1a-8b0030fbbc5c",
  "enduser.auth_type": "none",
  "event.date": "Tue, 21 Mar 2023",
  "level": "error",
  "message": "TelemetryEvent",
  "recording.isCypress": false,
  "recordingId": "a9233b5f-519c-4951-8233-0d6574407acc",
  "replay.recording.isCypress": false,
  "service_name": "telemetry-server",
  "telemetry.category": "ProtocolMessageCommandFailedCrashReport",
  "telemetry.data.buildId": "linux-chromium-20230321-1cd1ac564ffe-bfa354b141e5",
  "telemetry.data.buildMetadata.date": "20230321",
  "telemetry.data.buildMetadata.platform": "linux",
  "telemetry.data.buildMetadata.runtime": "chromium",
  "telemetry.data.controlId": "b9ab764b-5b04-4c56-9f1a-8b0030fbbc5c",
  "telemetry.data.crashReport.bucket": "",
  "telemetry.data.crashReport.bucketFixed": false,
  "telemetry.data.crashReport.buildId": "linux-chromium-20230321-1cd1ac564ffe-bfa354b141e5",
  "telemetry.data.crashReport.controlId": "b9ab764b-5b04-4c56-9f1a-8b0030fbbc5c",
  "telemetry.data.crashReport.description": "",
  "telemetry.data.crashReport.forTest": true,
  "telemetry.data.crashReport.linkerVersion": "linker-linux-10620-bfa354b141e5",
  "telemetry.data.crashReport.process.backtrace": {
    "frames": [
      "V8_Fatal(char.const*,....)+451",
      "v8::internal::ScopeIterator::HasContext().const+82",
      "v8::internal::DebugEvaluate::ContextBuilder::ContextBuilder(v8::internal::Isolate*,.v8::internal::JavaScriptFrame*,.int)+356",
      "v8::internal::DebugEvaluate::Local(v8::internal::Isolate*,.v8::internal::StackFrameId,.int,.v8::internal::Handle<v8::internal::String>,.bool)+224",
      "v8::internal::DebugStackTraceIterator::Evaluate(v8::Local<v8::String>,.bool)+97",
      "v8_inspector::V8DebuggerAgentImpl::evaluateOnCallFrame(v8_inspector::String16.const&,.v8_inspector::String16.const&,.v8_crdtp::detail::ValueMaybe<v8_inspector::String16>,.v8_crdtp::detail::ValueMaybe<bool>,.v8_crdtp::detail::ValueMaybe<bool>,.v8_crdtp::detail::ValueMaybe<bool>,.v8_crdtp::detail::ValueMaybe<bool>,.v8_crdtp::detail::ValueMaybe<bool>,.v8_crdtp::detail::ValueMaybe<double>,.std::Cr::unique_ptr<v8_inspector::protocol::Runtime::RemoteObject,.std::Cr::default_delete<v8_inspector::protocol::Runtime::RemoteObject>.>*,.v8_crdtp::detail::PtrMaybe<v8_inspector::protocol::Runtime::ExceptionDetails>*)+614",
      "v8_inspector::protocol::Debugger::DomainDispatcherImpl::evaluateOnCallFrame(v8_crdtp::Dispatchable.const&)+476",
      "v8_crdtp::UberDispatcher::DispatchResult::Run()+36",
      "v8_inspector::V8InspectorSessionImpl::dispatchProtocolMessage(v8_inspector::StringView)+474",
      "blink::SendCDPMessage(v8::FunctionCallbackInfo<v8::Value>.const&)+426",
      "v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo)+345",
      "v8::internal::MaybeHandle<v8::internal::Object>.v8::internal::(anonymous.namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*,.v8::internal::Handle<v8::internal::HeapObject>,.v8::internal::Handle<v8::internal::FunctionTemplateInfo>,.v8::internal::Handle<v8::internal::Object>,.unsigned.long*,.int)+1013",
      "v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments,.v8::internal::Isolate*)+296",
      "0x100a3ff15b38",
      "0x100a3fe8b82c",
      "0x100a3fe8b82c",
      "0x100a3fe8b82c",
      "0x100a3fe8b82c",
      "0x100a3fe89e5c",
      "0x100a3fe89b87",
      "v8::internal::(anonymous.namespace)::Invoke(v8::internal::Isolate*,.v8::internal::(anonymous.namespace)::InvokeParams.const&)+2419",
      "v8::internal::Execution::Call(v8::internal::Isolate*,.v8::internal::Handle<v8::internal::Object>,.v8::internal::Handle<v8::internal::Object>,.int,.v8::internal::Handle<v8::internal::Object>*)+284",
      "v8::internal::CommandCallback(char.const*,.char.const*)+658",
      "recordreplay::SendCommand(char.const*,.Json::Value.const&,.bool)+426",
      "EvaluateInFrameCommand::Process(Json::Value.const&,.std::__cxx11::basic_string<char,.std::char_traits<char>,.std::allocator<char>.>&)+132",
      "CommandRequest::Process(Json::Value.const&)+516",
      "recordreplay::PerformPauseDataRequest(Json::Value.const&)+276",
      "PauseRequestHandler::Start(Json::Value.const&)+210",
      "ManifestFinished(Json::Value*)+751",
      "RunToPointHandler::ReachedDestination(Json::Value.const&)+491",
      "recordreplay::OnScanTargetHit()+36",
      "recordreplay::OnInstrument(char.const*,.char.const*,.int)+1523",
      "v8::internal::OnInstrumentation(v8::internal::Isolate*,.v8::internal::Handle<v8::internal::JSFunction>,.int)+208",
      "v8::internal::Runtime_RecordReplayInstrumentation(int,.unsigned.long*,.v8::internal::Isolate*)+270",
      "0x100a3ff15a38",
      "0x100a3fffad11",
      "0x100a3fe8b82c",
      "0x100a3fe8b82c",
      "0x100a3fe8b82c",
      "0x100a3fe89e5c",
      "0x100a3fe89b87",
      "v8::internal::(anonymous.namespace)::Invoke(v8::internal::Isolate*,.v8::internal::(anonymous.namespace)::InvokeParams.const&)+2419",
      "v8::internal::Execution::Call(v8::internal::Isolate*,.v8::internal::Handle<v8::internal::Object>,.v8::internal::Handle<v8::internal::Object>,.int,.v8::internal::Handle<v8::internal::Object>*)+284",
      "v8::Function::Call(v8::Local<v8::Context>,.v8::Local<v8::Value>,.int,.v8::Local<v8::Value>*)+648",
      "blink::V8ScriptRunner::CallFunction(v8::Local<v8::Function>,.blink::ExecutionContext*,.v8::Local<v8::Value>,.int,.v8::Local<v8::Value>*,.v8::Isolate*)+915",
      "blink::bindings::CallbackInvokeHelper<blink::CallbackFunctionBase,.(blink::bindings::CallbackInvokeHelperMode)2,.(blink::bindings::CallbackReturnTypeIsPromise)0>::Call(int,.v8::Local<v8::Value>*)+91",
      "blink::V8EventHandlerNonNull::InvokeWithoutRunnabilityCheck(blink::bindings::V8ValueOrScriptWrappableAdapter,.blink::HeapVector<blink::ScriptValue,.0u>.const&)+365",
      "blink::JSEventHandler::InvokeInternal(blink::EventTarget&,.blink::Event&,.v8::Local<v8::Value>)+1326",
      "blink::JSBasedEventListener::Invoke(blink::ExecutionContext*,.blink::Event*)+744",
      "blink::EventTarget::FireEventListeners(blink::Event&,.blink::EventTargetData*,.blink::HeapVector<blink::RegisteredEventListener,.1u>&)+1851",
      "blink::EventTarget::FireEventListeners(blink::Event&)+361",
      "blink::EventTarget::DispatchEventInternal(blink::Event&)+64",
      "blink::XMLHttpRequest::DispatchReadyStateChangeEvent()+680",
      "blink::XMLHttpRequest::EndLoading()+205",
      "blink::XMLHttpRequest::DidFinishLoadingInternal()+328",
      "blink::XMLHttpRequest::DidFinishLoading(unsigned.long)+223",
      "blink::ThreadableLoader::NotifyFinished(blink::Resource*)+266",
      "blink::Resource::NotifyFinished()+231",
      "blink::ResourceFetcher::HandleLoaderFinish(blink::Resource*,.base::TimeTicks,.blink::ResourceFetcher::LoaderFinishType,.unsigned.int,.bool)+769",
      "blink::ResourceLoader::DidFinishLoading(base::TimeTicks,.long,.long,.long,.bool,.absl::optional<bool>)+839",
      "non-virtual.thunk.to.blink::ResourceLoader::DidFinishLoadingBody()+96",
      "blink::ResponseBodyLoader::OnStateChange()+1311",
      "blink::ResourceLoader::DidFinishLoading(base::TimeTicks,.long,.long,.long,.bool,.absl::optional<bool>)+934",
      "blink::WebURLLoader::Context::OnCompletedRequest(network::URLLoaderCompletionStatus.const&)+362",
      "blink::WebResourceRequestSender::OnRequestComplete(network::URLLoaderCompletionStatus.const&)+763",
      "blink::MojoURLLoaderClient::FlushDeferredMessages()+475",
      "blink::MojoURLLoaderClient::StoreAndDispatch(std::Cr::unique_ptr<blink::MojoURLLoaderClient::DeferredMessage,.std::Cr::default_delete<blink::MojoURLLoaderClient::DeferredMessage>.>)+173",
      "blink::ThrottlingURLLoader::OnComplete(network::URLLoaderCompletionStatus.const&)+504",
      "network::mojom::URLLoaderClientStubDispatch::Accept(network::mojom::URLLoaderClient*,.mojo::Message*)+605",
      "mojo::InterfaceEndpointClient::HandleValidatedMessage(mojo::Message*)+753",
      "mojo::MessageDispatcher::Accept(mojo::Message*)+211",
      "mojo::InterfaceEndpointClient::HandleIncomingMessage(mojo::Message*)+58",
      "mojo::internal::MultiplexRouter::ProcessIncomingMessage(mojo::internal::MultiplexRouter::MessageWrapper*,.mojo::internal::MultiplexRouter::ClientCallBehavior,.base::SequencedTaskRunner*)+1022",
      "mojo::internal::MultiplexRouter::Accept(mojo::Message*)+363",
      "mojo::MessageDispatcher::Accept(mojo::Message*)+211",
      "mojo::Connector::DispatchMessage(mojo::ScopedHandleBase<mojo::MessageHandle>)+471",
      "mojo::Connector::ReadAllAvailableMessages()+244",
      "base::RepeatingCallback<void.(int)>::Run(int).const.&+39",
      "mojo::SimpleWatcher::OnHandleReady(int,.unsigned.int,.mojo::HandleSignalsState.const&)+291",
      "base::internal::Invoker<base::internal::BindState<void.(mojo::SimpleWatcher::*)(int,.unsigned.int,.mojo::HandleSignalsState.const&),.base::WeakPtr<mojo::SimpleWatcher>,.int,.unsigned.int,.mojo::HandleSignalsState>,.void.()>::RunOnce(base::internal::BindStateBase*)+142",
      "base::TaskAnnotator::RunTaskImpl(base::PendingTask&)+257",
      "base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(base::LazyNow*)+965",
      "base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()+139",
      "non-virtual.thunk.to.base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()+21",
      "base::MessagePumpDefault::Run(base::MessagePump::Delegate*)+154",
      "base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool,.base::TimeDelta)+355",
      "base::RunLoop::Run(base::Location.const&)+461",
      "content::RendererMain(content::MainFunctionParams)+1434",
      "content::RunOtherNamedProcessTypeMain(std::Cr::basic_string<char,.std::Cr::char_traits<char>,.std::Cr::allocator<char>.>.const&,.content::MainFunctionParams,.content::ContentMainDelegate*)+686",
      "content::ContentMainRunnerImpl::Run()+637",
      "content::RunContentProcess(content::ContentMainParams,.content::ContentMainRunner*)+294",
      "content::ContentMain(content::ContentMainParams)+95",
      "ChromeMain+525",
      "new_libc_start_main(void.(*)(int,.char.const**))+22",
      "_start+42",
      "onstack:V8_Fatal(char.const*,....)+451",
      "onstack:v8::internal::Object::TransitionAndWriteDataProperty(v8::internal::LookupIterator*,.v8::internal::Handle<v8::internal::Object>,.v8::internal::PropertyAttributes,.v8::Maybe<v8::internal::ShouldThrow>,.v8::internal::StoreOrigin)+303",
      "onstack:v8::internal::Object::SetPropertyOrElement(v8::internal::Isolate*,.v8::internal::Handle<v8::internal::Object>,.v8::internal::Handle<v8::internal::Name>,.v8::internal::Handle<v8::internal::Object>,.v8::Maybe<v8::internal::ShouldThrow>,.v8::internal::StoreOrigin)+309",
      "onstack:bool.std::Cr::__function::__policy_invoker<bool.(v8::internal::Handle<v8::internal::String>,.v8::internal::Handle<v8::internal::Object>,.v8::internal::ScopeIterator::ScopeType)>::__call_impl<std::Cr::__function::__default_alloc_func<v8::internal::ScopeIterator::ScopeObject(v8::internal::ScopeIterator::Mode)::$_1,.bool.(v8::internal::Handle<v8::internal::String>,.v8::internal::Handle<v8::internal::Object>,.v8::internal::ScopeIterator::ScopeType)>.>(std::Cr::__function::__policy_storage.const*,.v8::internal::Handle<v8::internal::String>&&,.v8::internal::Handle<v8::internal::Object>&&,.v8::internal::ScopeIterator::ScopeType)+194",
      "onstack:v8::internal::ScopeIterator::VisitLocals(std::Cr::function<bool.(v8::internal::Handle<v8::internal::String>,.v8::internal::Handle<v8::internal::Object>,.v8::internal::ScopeIterator::ScopeType)>.const&,.v8::internal::ScopeIterator::Mode,.v8::internal::ScopeIterator::ScopeType).const+468",
      "onstack:v8::internal::FrameInspector::GetExpression(int)+61",
      "onstack:v8::internal::ScopeIterator::VisitLocalScope(std::Cr::function<bool.(v8::internal::Handle<v8::internal::String>,.v8::internal::Handle<v8::internal::Object>,.v8::internal::ScopeIterator::ScopeType)>.const&,.v8::internal::ScopeIterator::Mode,.v8::internal::ScopeIterator::ScopeType).const+236",
      "onstack:v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller(v8::internal::Handle<v8::internal::Map>,.int,.v8::internal::Handle<v8::internal::Oddball>,.v8::internal::AllocationType)+68",
      "onstack:chrome+22405751",
      "onstack:v8::internal::Factory::NewSlowJSObjectFromMap(v8::internal::Handle<v8::internal::Map>,.int,.v8::internal::AllocationType,.v8::internal::Handle<v8::internal::AllocationSite>)+83",
      "onstack:v8::internal::ScopeIterator::HasContext().const+82",
      "onstack:v8::internal::DebugEvaluate::ContextBuilder::ContextBuilder(v8::internal::Isolate*,.v8::internal::JavaScriptFrame*,.int)+356",
      "onstack:v8::internal::DebugEvaluate::Local(v8::internal::Isolate*,.v8::internal::StackFrameId,.int,.v8::internal::Handle<v8::internal::String>,.bool)+224",
      "onstack:new_pthread_mutex_lock(void*)+21",
      "onstack:chrome+222086992",
      "onstack:chrome+222087112",
      "onstack:chrome+222087232",
      "onstack:chrome+222088568",
      "onstack:chrome+222089584",
      "onstack:chrome+222089704",
      "onstack:chrome+222088808",
      "onstack:chrome+222088928",
      "onstack:chrome+222088688",
      "onstack:InstructionPoint"
    ],
    "threadId": 1
  },
  "telemetry.data.crashReport.process.manifest": {
    "kind": "pauseRequest",
    "requestMethod": "Pause.evaluateInFrame",
    "requestType": "command"
  },
  "telemetry.data.crashReport.process.text": "Check failed: needs_context && current_scope_ == closure_scope_ && current_scope_->is_function_scope() && !function_->is_null() implies function_->context() != *context_.",
  "telemetry.data.crashReport.process.threadId": 1,
  "telemetry.data.crashReport.process.why": "LinkerApplicationCrash",
  "telemetry.data.crashReport.processId": 108,
  "telemetry.data.crashReport.processNonFatal.threadId": 0,
  "telemetry.data.crashReport.processNonFatal.why": "MissingReport",
  "telemetry.data.crashReport.recordingAssertionsDisabled": false,
  "telemetry.data.crashReport.recordingFeaturesDisabled": "",
  "telemetry.data.crashReport.recordingId": "a9233b5f-519c-4951-8233-0d6574407acc",
  "telemetry.data.crashReport.sessionIds": [
    "b9ab764b-5b04-4c56-9f1a-8b0030fbbc5c/7dd15050-90b1-4cb7-9b3d-897d3d1b2326"
  ],
  "telemetry.data.crashReport.why": "NonFatal",
  "telemetry.data.crashReportUuid": "1fd4de0a-6ef6-4afb-9c43-073ee23985cd",
  "telemetry.data.event": "undefined",
  "telemetry.data.forTest": true,
  "telemetry.data.protocolMessageId": 267,
  "telemetry.data.recording.internal": true,
  "telemetry.data.recording.isCypress": false,
  "telemetry.data.recording.userEmail": "internal-tests@replay.io",
  "telemetry.data.recording.userPriority": 5,
  "telemetry.data.recording.workspaceName": "Replay Internal Livetests",
  "telemetry.data.recordingId": "a9233b5f-519c-4951-8233-0d6574407acc",
  "telemetry.data.requestCommandMethod": "Pause.evaluateInFrame",
  "telemetry.data.requestOptions.shortTimeout": true,
  "telemetry.data.requestType": "command",
  "telemetry.data.service_name": "controller",
  "telemetry.data.sessionId": "b9ab764b-5b04-4c56-9f1a-8b0030fbbc5c/7dd15050-90b1-4cb7-9b3d-897d3d1b2326",
  "telemetry.data.success": "n/a",
  "telemetry.data.timestamp": 1679396489198,
  "telemetry.userData": "undefined",
  "userEmail": "internal-tests@replay.io"
}

@bhackett1024

The PR looks fine, I don't know why the command crashes are happening and I'm not really sure what you're asking for. Looking at that stack trace there is V8_Fatal on the stack so V8 is crashing intentionally for some reason and I'd look into why. From the stack this is also happening during an evaluateInFrame command, have you tried manually testing that?

That evaluateInFrame is part of the initial runToPoint, and I did test this locally, and I re-tested it online, and it did not show up.
The weird part is that in that one test run it happened so consistently, and now its just gone, despiste both runs having the same buildId. Very odd.

@bhackett1024 Yeah, I'd love to know who is sending it. I have looked into this in the past for a little while, but have not found out yet who sends the commands.

repo: replayio/chromium


text:
Support for preleases

Currently, a kots prerelease will always use master (alpha image tag) of kotsadm. We intentionally ignore prerelease tags and always use alpha. IIRC, we do this so that interim releases work, but this is starting to make beta releases difficult. The code is here: https://github.com/replicatedhq/kots/blob/5dd232b0e5361630f0cc0ed441151749087d0ae3/pkg/kotsadm/kotsadm_version.go#L30
There doesn't seem to be a test preventing us from changing this. @divolgin or @laverya do you know why this code exists?

This is added in 1.14.0 (and being tested with 1.14.0-beta.2 tag)

repo: replicatedhq/kots


text:
Stateless not working

I am using stateless for entity (see previous issue #274) but it doesn't appear to be working. The booleans aren't getting updated when I call update. Any ideas?
@Entity(stateless = true)
abstract class AbstractFriendship {

    @Key
    String mParseId;

    @Convert(ConverterUser.class)
    User mPerson1;
    Boolean mConfirm1;
    @Convert(ConverterUser.class)
    User mPerson2;
    Boolean mConfirm2;
    Date mFriendsSince;

}

Which update API are you calling? if it's a non-blocking API you may need to evaluate the call further (eg subscribe if its RxJava).

The update call
Application.getData().update(friendship);
Code from Application.class
public static EntityDataStore<Persistable> getData() {
        if (sDataStore == null) {
            final EntityModel model = Models.DEFAULT;
            DatabaseSource source = new DatabaseSource(instance, model, 1);
            if (BuildConfig.DEBUG) {
                // use this in development mode to drop and recreate the tables on every upgrade
                source.setTableCreationMode(TableCreationMode.DROP_CREATE);
            }
            Configuration configuration = source.getConfiguration();
            sDataStore = new EntityDataStore<>(configuration);
        }
        return sDataStore;
    }

I am using rc4.

Added a specific test case for this in this commit and it works fine, the update statement is correct.
INFO: beforeExecuteUpdate org.sqlite.jdbc4.JDBC4PreparedStatement@3bd40a57 sql: update Entry set id = ? ,created = ? ,flag1 = ? ,flag2 = ? where id = ?

Yep I am seeing a similar statement
09-21 09:34:29.390 30333-30431/com.example.dev I/requery: beforeExecuteUpdate sql: update Friendship set mParseId = ? ,mConfirm1 = ? ,mConfirm2 = ? ,mFriendsSince = ? ,mPerson1 = ? ,mPerson2 = ? where mParseId = ?
09-21 09:34:29.392 30333-30431/com.example.dev I/requery: afterExecuteUpdate

However it is still not getting set. I am watching it in debug and I am changing the value from true to false but the database entry is not getting updated. Could it be anything to do with the fact that I have Converters in the friendship object maybe?

I would check you are updating the right row with the key you are expecting (mParseId) and also that something else isn't overwriting or changing the values in your object.

As far as I can tell, I am not overwriting my change. I am running through in debug and checking nothing is overriding. I am creating the friendship object with a constructor I have written in the abstract entity class. Could that be having any affect?

No the other objects shouldn't affect those fields

Sorry I imagine you're very busy but have you got any other ideas for me. I'm struggling to find the problem.

So I have found a work around. If I disable the cache on the object everything works as expected. Does that give you any hint as to what the problem might be?

Disabling the cache mean's objects don't get reused, possibly the original cached object was modified by code elsewhere in your app?

Closing for now, please post back if you get more info thanks

authors: alistairsykes npurushe

repo: requery/requery


text:
./examples/python/text_logging/main.py is broken

Nothing shows up when running it.
./examples/python/dicom/main.py also sporadically fails.

Duplicate of https://github.com/rerun-io/rerun/issues/2124

repo: rerun-io/rerun


text:
fix(intl): expose Intl.Common types

The Intl.Common types don't seem to be exposed. And while they technically don't need to be, since they're structural, it's useful to be able to reference them for derived APIs (as I have in a project of mine).
It might be more consistent to expose the types directly in relevant modules though, instead of the module as a whole, but this is the minimal change needed.

@glennsl perfect, thank you for cleaning up as well! Would you like a 1.1.1 with this so you can use this in your project right away? Easily done if so.

Thanks, but no need for my sake. I found the zeroTo20 and zeroTo21 types a bit too restrictive amyway, so I just continued to use ints and sprinkled some magic on them instead.

authors: glennsl zth

repo: rescript-association/rescript-core


text:
Question: Would it be possible to make the key salt accessible?

While I analyzed the encryption scheme of SQLCipher, I found out that SQLCipher stores the key salt in the first 16 bytes of the database header just as sqleet. However, SQLCipher does not include the key salt in the HMAC calculation of database page 1, while sqleet does.
In the end it seems to be irrelevant whether the key salt is included in the HMAC calculation or not. If the key salt bytes in the first 16 bytes of the database file are modified - either accidentally or on purpose -, this will make the database file inaccessible.
The problem I see is that there is almost no chance to recover a database corrupted in such a way, since the user only knows the passphrase he used, but he does not know the random key salt.
IMHO it would make sense to implement an option to access the generated key salt, so that it could be stored in a safe place, and/or to let the user set a specific key salt instead of generating a random one.
This would allow to create a tool to recover a corrupted encrypted database - at least partially.

My initial reaction was that accessing the salt is so rare operation that it does not justify adding a C API fr it. Especially since competent people dealing with data corruption etc. can already access
accessing the salt would be almost never needed so offering C API for it would just introduce unnecessary complexity. Moreover, competent people dealing with data corruption

Sorry about the accidental close...
My initial reaction was that adding a C API for accessing the salt would be unjustified because it is so rare operation. Moreover, competent people with hex editors can already access and modify the salt quite easily because it is stored in the first 16 bytes of the database file, in cleartext. Average users should not have to care about their salts (except for fixing data corruption -- see my response to those issues in the end of this comment).
However, I have started to think that we need to have some control of the salt but for another reason. SQLCipher (and other libraries too?) support specifying a raw key which bypasses the key derivation function. I like the idea because it allows program developers, who use sqleet as a library, to take a full responsibility of the key derivation process. This would address the problem that sqleet's PBKDF2-HMAC-SHA256 is far from ideal, especially since I want to keep the configuration of KDF parameters strictly compile-tome only (for religious design reasons).
Support for raw keys has the consequence that sqleet no longers needs the salt, while the external program handling key derivation might do. Hence, I have been considering something like this:
diff --git a/README.md b/README.md
index d8cd6af..5fa7c53 100644
--- a/README.md
+++ b/README.md
@@ -135,6 +135,15 @@ In addition, there are `sqlite3_key_v2()` and `sqlite3_rekey_v2()` functions
 that accept the target database name as the second parameter. By default, the
 main database is used.

+The above functions pass the provided key string (password) to a key derivation
+algorithm (i.e., PBKDF2-HMAC-SHA256 with a 16-byte salt and 12345 iterations).
+Optionally, the user can bypass the key derivation by specifying a raw key in
+format `raw:X` where `X` is a 32-byte binary string or a 64-digit hex-encoded
+string. This can be useful in programs that use sqleet as a library and want to
+handle key derivation by themselves. Additionally, the raw key string can also
+be followed by 16-byte (or 32-hexdigit) salt which is stored in the beginning
+of the database file.
+

 Android support
 ---------------
diff --git a/sqleet.c b/sqleet.c
index 06e6561..3695a83 100644
--- a/sqleet.c
+++ b/sqleet.c
@@ -42,10 +42,62 @@ Codec *codec_dup(Codec *src)
     return codec;
 }

+static int hex_decode(const char *hex, unsigned int n, char *out)
+{
+    int i;
+    for (i = 0; i < n; i++) {
+        char c = hex[i];
+        if (c >= '0' && c <= '9') {
+            c = c - '0';
+        } else if (c >= 'A' && c <= 'F') {
+            c = c - 'A' + 10;
+        } else if (c >= 'a' && c <= 'f') {
+            c = c - 'a' + 10;
+        } else {
+            int j = (i+1) / 2;
+            for (i = 0; i < j; i++) {
+                ((volatile char *)out)[i] = 0;
+            }
+            return 0;
+        }
+        out[i/2] = (out[i/2] << 4) | c;
+    }
+    return 1;
+}
+
 void codec_kdf(Codec *codec)
 {
+    /* Skip key derivation if the given key starts with "raw:" */
+    if (codec->nKey > 4 && !memcmp(codec->zKey, "raw:", 4)) {
+        switch (codec->nKey) {
+        /* Binary key (and salt) */
+        case 4 + 32 + 16:
+            memcpy(codec->salt, (const char*)codec->zKey + 4 + 32, 16);
+            /* fall-through */
+        case 4 + 32:
+            memcpy(codec->key, (const char*)codec->zKey + 4, 32);
+            goto done;
+
+        /* Hex-encoded key (and salt) */
+        case 4 + 64 + 32:
+            if (!hex_decode((const char*)codec->zKey + 4 + 64, 32, codec->salt))
+                break;
+            /* fall-through */
+        case 4 + 64:
+            if (!hex_decode((const char*)codec->zKey + 4, 64, codec->key))
+                break;
+            goto done;
+
+        default:
+            break;
+        }
+    }
+
+    /* Run key-derivation algorithm on the key string */
     pbkdf2_hmac_sha256(codec->zKey, codec->nKey, codec->salt, 16, 12345,
                        codec->key, 32);
+
+done:
     codec->zKey = NULL;
     codec->nKey = 0;
 }

The patch would allow specifying a raw key and, optionally, a salt for storing it in the first 16 bytes of the database. The proposed patch does not specially support key derivation from a passphrase with a specified salt. However, I think this would still provide at least a partial solution to the " corrupted salt" problem: the user could perform KDF by himself using PBKDF2-HMAC-SHA256 with 12345 iterations and any salt he wants, which then could be tried to access the database.

The problem I see is that there is almost no chance to recover a database corrupted in such a way, since the user only knows the passphrase he used, but he does not know the random key salt.
IMHO it would make sense to implement an option to access the generated key salt, so that it could be stored in a safe place, and/or to let the user set a specific key salt instead of generating a random one.
This would allow to create a tool to recover a corrupted encrypted database - at least partially.

... To be written. Sauna break ->

My initial reaction was that adding a C API for accessing the salt would be unjustified because it is so rare operation.

It might be a rare operation, because in fact it would be only necessary on creating a new database, for example to be able to save the salt in a safe place for recovery in case of database corruption.

Moreover, competent people with hex editors can already access and modify the salt quite easily because it is stored, cleartext, in the first 16 bytes of the database file.

And that's exactly why I find it at least useful to offer a safe API method instead of forcing a user to use a hex editor with which one can easily corrupt the database accidentally. A single modified salt bit is enough to corrupt the database and make it inaccessible.

Average users should not have to care about salts at all (except for maybe fixing data corruption -- see my response to those issues in the end of this comment).

A user knows the key (password phrase) per definition, but he doesn't know the random salt. That is, in case of a corrupted salt the user has almost no chance to recover his database.

However, I have started to think that we need to have some control of the salt but for another reason. SQLCipher (and other libraries too?) support specifying a raw key which bypasses the key derivation function.

SQLCipher definitely allows to specify key and/or salt as a raw key without using a key derivation function. I don't know about other libraries.

I like the idea because it allows program developers, who use sqleet as a library, to take full responsibility of the key derivation process. This would address the problem that sqleet's PBKDF2-HMAC-SHA256 is far from ideal, especially since I want to keep the configuration of KDF parameters strictly compile-time only (for religious design reasons).

The reasoning is somewhat illogical. On the one hand you want to forbid that a user changes configuration parameters at runtime, on the other hand you want to allow complete freedom at runtime. Why not allow changing configuration parameters at runtime as a compromise?

Raw key support has the consequence that sqleet no longer needs the salt (when raw keys are being used), whereas the external program handling the key derivation might do. Hence, I have been considering something like this: [...]

Looks ok for me.

Data corruption (due to hardware failure or whatever) is a major issue and needs to be addressed,

Right. And exactly for this purpose you will need the salt, because ...

but IMO preferably with special tooling instead of tackling potential issues at library-level in sqleet.

... whatever tools you develop, without an intact salt you are fighting a lost battle, if the salt was affected by the data corruption.

I think the way to go is to write a bunch of Python scripts under scripts/ directory in the repository, which handle, e.g., key derivation, full encrypts and decrypts, potential version migrations, as well as dumping and manipulating internal information such as the database salt, nonces and Poly1305 tags. Most importantly, the scripts should support something like forcibly decrypting a given database ignoring any MAC errors etc.

To be able to try to recover a corrupted database the latter is definitely necessary.

Scripts are the next thing on my TODO list after the raw key support ...
TLDR Raw key support will help a bit in the near future, but external scripts are needed for proper data rescue and salt manipulation.

IMHO there is nothing wrong in adding methods to the library itself for those purposes.

Commit 3d2c17b adds raw keys and support for specifying a salt. I'm leaving this issue open until we implement a way to read the salt (be it a C function or a Python script).

I like the idea because it allows program developers, who use sqleet as a library, to take full responsibility of the key derivation process.

The reasoning is somewhat illogical. On the one hand you want to forbid that a user changes configuration parameters at runtime, on the other hand you want to allow complete freedom at runtime. Why not allow changing configuration parameters at runtime as a compromise?

I do not want the average user to select crypto parameters because it introduces the risk of insecure configurations and compatibility issues (opening a database created with unknown parameters would be difficult even if knowing the correct key). Compile-time configuration is OK because it requires some effort to modify, so mostly advanced users who know what they are doing will use that for good reasons. I think that raw keys, although easily misused, fall into the latter category.

Commit 3d2c17b adds raw keys and support for specifying a salt.

I will incorporate this option into my sqleet variant.

I'm leaving this issue open until we implement a way to read the salt (be it a C function or a Python script).

I would prefer a C function as an extension to the API.

I do not want the average user to select crypto parameters because it introduces the risk of insecure configurations and compatibility issues (opening a database created with unknown parameters would be difficult even if knowing the correct key). Compile-time configuration is OK because it requires some effort to modify, so mostly advanced users who know what they are doing will use that for good reasons. I think that raw keys, although easily misused, fall into the latter category.

My experience is that the average user doesn't tamper with any configuration options, neither at compile time nor at runtime. Offering a raw key/salt option makes it extremely easy for such users to misuse this option. However, since libraries like SQLCipher offer raw keys, adding this option to sqleet as well is certainly reasonable.
For the experienced user it may be convenient to be able to use configuration options at runtime as an alternative ro implementing the complete key derivation process himself. However, I respect your decision not to follow this route.

I'd like to get back to this issue about accessing the key salt.
Just recently SQLCipher 4.0.0 was released. In addition to now using SHA512 for key derivation and HMAC a new pragma cipher_plaintext_header_size was introduced. This new pragma addresses SQLCipher issue 255 which is related to a certain iOS use case where iOS applies special handling if an application uses a SQLite database in WAL mode. However, iOS identifies SQLite database files by inspecting the first 16 bytes of the database header and looking for the SQLite format identification string (SQLite format 3\000), and therefore these bytes have to be unencrypted. In default operation mode SQLCipher uses the first 16 bytes of the database header for storing the key salt - just as sqleet. However, if the new pragma is applied, the key salt can't be stored any longer in the database file. Instead the user has to retrieve the key salt using another pragma (cipher_salt), and to set in each time on opening the database.
I can't judge how important it would be to support this iOS use case, but obviously the SQLCipher team found it important enough to introduce additional pragmas.
If one wants to support this iOS use case then IMHO it would be necessary to provide methods to retrieve and set the key salt independently from the key itself. Currently sqleet supports only a combination of raw key plus salt.
What's your opinion?

Hmm... That's an interesting problem that I was unaware of. Thanks for notifying me.
AFAIK the iOS use-case of sqleet is not very popular and there already exist (poor) workarounds, so the issue is not critical. I'd prefer not to add a new PRAGMA like SQLCipher mainly due to the maintenance burden (IIRC SQLite3 offers no public/stable API to add custom PRAGMAs). Cloning sqleet master and replacing old SQLite3 amalgamation with the latest version should ideally work "out-of-the-box" (so far this has been true except for a couple rekeyvacuum.c updates and added sqlite3BtreeBeginTrans() NULL parameter in 3.25.1). The recent SQLite3 <v3.26.0 RCE vulnerability shows how important this "feature" is in security-sensitive applications.
I think there are other feasible solutions. For example, a more powerful interface for specifying salt and key using existing PRAGMAs:
PRAGMA key='header:SQLite format 3\x00'     # File header
PRAGMA key='salt:cipher_salt_of_any_length' # PBKDF2 salt (not stored in the header)
PRAGMA key='password_for_key_derivation'    # PBKDF2 password

I agree that setting PRAGMA key three times can look a bit confusing. Alternatively we could provide the various parameters via query parameters of URI filenames. This would also have the advantage of automatically handling URI-encoded hex values in a standard way. However, escaping of special characters (such as &) in query parameter values could become a problem.
Another approach would be to write a compatibility-VFS shim for iOS (see The SQLite OS Interface or "VFS", and in particular appendvfs.c).
There are probably more solutions as well. I will have to think more about this issue before deciding which approach to take. Your (and everyone else's) input is welcome, as always.

AFAIK the iOS use-case of sqleet is not very popular and there already exist (poor) workarounds, so the issue is not critical.

AFAICT this iOS issue affects all SQLite encryption extensions (even the official one), if the database file signature (first 16 bytes) plus the first part of the database header (next 8 bytes) are not in plain text.
I have no idea whether anybody is actually using my own SQLite encryption extension (wxSQLite3) in an iOS environment and whether this issue imposes a real problem. At least up to now no one contacted me on this behalf.
So, I agree that it seems not to be a critical issue.

I'd prefer not to add a new PRAGMA like SQLCipher mainly due to the maintenance burden (IIRC SQLite3 offers no public/stable API to add custom PRAGMAs).

Indeed, adding custom PRAGMAs requires changes to the SQLite source code. And I, too, prefer to not touch the original SQLite code.
However, it is not very difficult to accomplish the tasks of a PRAGMA command using a SELECT command and adding a user-defined function. This is the approach I chose for wxSQLite3.

Cloning sqleet master and replacing old SQLite3 amalgamation with the latest version should ideally work "out-of-the-box" (so far this has been true except for a couple rekeyvacuum.c updates and added sqlite3BtreeBeginTrans() NULL parameter in 3.25.1). The recent SQLite3 <v3.26.0 RCE vulnerability shows how important this "feature" is in security-sensitive applications.

I fully agree. Being able to update the underlying SQLite implementation easily and fast is critical.

I think there are other feasible solutions. For example, a more powerful interface for specifying salt and key using existing PRAGMAs:
PRAGMA key='header:SQLite format 3\x00'     # File header
PRAGMA key='salt:cipher_salt_of_any_length' # PBKDF2 salt (not stored in the header)
PRAGMA key='password_for_key_derivation'    # PBKDF2 password

Hm, yes, it could be done that way, but this is sort of hijacking PRAGMA key=... and could easily cause mistakes in using it. For example, this approach requires some extra housekeeping, because one has to remember all pieces of information until the underlying codec is actually attached to the database - that is, the order of the PRAGMAs could matter. Of course, the housekeeping can't be avoided even with different PRAGMA names, but it would be a bit less errorprone.

I agree that setting PRAGMA key three times can look a bit confusing. Alternatively we could provide the various parameters via query parameters of URI filenames. This would also have the advantage of automatically handling URI-encoded hex values in a standard way. However, escaping of special characters (such as &) in query parameter values could become a problem.

In the latest version of wxSQLite3 I added support for passing codec parameters via URI query parameters. This works very smoothly and is less cumbersome than using a bunch of PRAGMAs (or SELECTs with user-defined functions). IMHO it should therefore be the preferred method, because it eliminates the need for extra housekeeping - all URI parameters are directly accessible when attaching the codec.
IMHO escaping special characters is an almost neglectable problem. Or at least, it doesn't impose a new problem, because one has to escape certain special characters, too, when using a PRAGMA. Regarding raw key and salt values it would be best to pass them as hex values anyway.

Another approach would be to write a compatibility-VFS shim for iOS (see The SQLite OS Interface or "VFS", and in particular appendvfs.c).

At the moment I don't see how this would help to overcome the mentioned iOS issue (where iOS simply looks at the database file itself without using any SQLite code). The best solution to the iOS issue would be to change the way how iOS identifies SQLite database files. However, this is way beyond our scope.

There are probably more solutions as well. I will have to think more about this issue before deciding which approach to take. Your (and everyone else's) input is welcome, as always.

For the SQLCipher support in wxSQLite3 I added measures to handle the database format variant that was introduced to overcome the iOS issue. However, I didn't add any special handling yet for the other cipher schemes supported by wxSQLite3.
While implementing support for the latest SQLCipher version I stumbled across other issues. For example, SQLCipher adds its own memory handling methods. These methods do 2 things: (1) allocated SQLite memory is locked, so that the OS doesn't swap it to disk, and (2) freed SQLite memory is overwritten with zeros. I doubt that (1) really works under all circumstances, since the OS usually limits the amount of memory that can be locked.

Hm, yes, it could be done that way, but this is sort of hijacking PRAGMA key=... and could easily cause mistakes in using it. For example, this approach requires some extra housekeeping, because one has to remember all pieces of information until the underlying codec is actually attached to the database - that is, the order of the PRAGMAs could matter

The extra housekeeping does not necessarily require too much extra logic if implemented carefully and the basic usage would remain simple, i.e., only setting the key once. The PRAGMA ordering is indeed mistake-prone, but IMO the order requirements are intuitive enough (the only thing that matters is setting the password after other settings). However, I'm still unsure about how rekey should work under this model.

However, it is not very difficult to accomplish the tasks of a PRAGMA command using a SELECT command and adding a user-defined function. This is the approach I chose for wxSQLite3.

Sounds reasonable. I will have to look into this. Maintaining UX compatibility between sqleet and wxSQLite3 would also be nice.

In the latest version of wxSQLite3 I added support for passing codec parameters via URI query parameters. This works very smoothly and is less cumbersome than using a bunch of PRAGMAs (or SELECTs with user-defined functions). IMHO it should therefore be the preferred method, because it eliminates the need for extra housekeeping - all URI parameters are directly accessible when attaching the codec.
IMHO escaping special characters is an almost neglectable problem. Or at least, it doesn't impose a new problem, because one has to escape certain special characters, too, when using a PRAGMA. Regarding raw key and salt values it would be best to pass them as hex values anyway.

... and this sounds even more reasonable. :)

Another approach would be to write a compatibility-VFS shim for iOS (see The SQLite OS Interface or "VFS", and in particular appendvfs.c).

At the moment I don't see how this would help to overcome the mentioned iOS issue (where iOS simply looks at the database file itself without using any SQLite code)

The idea is to use a VFS shim to add an extra SQLite3 format 3\x00 header in the beginning of the encrypted database file. The VFS shim could then redirect SQLite3's reads & writes by an offset of +16, basically skipping the fake header. As a result, iOS would recognize the database file as a SQLite3 database, but from the perspective of SQLite3/sqleet, the file would look like a regular encrypted database when accessed through the VFS shim.
The 16-byte offset may incur a noticeable performance cost though. Extending the fake header to database page boundary might help in that regard (or not).
But yeah, this feels like a dirty iOS-specific hack and not a proper solution.

The best solution to the iOS issue would be to change the way how iOS identifies SQLite database files. However, this is way beyond our scope.

I agree. The goal here is to improve the library interface in general instead of introducing special logic for supporting broken platforms which do weird things like iOS.

While implementing support for the latest SQLCipher version I stumbled across other issues. For example, SQLCipher adds its own memory handling methods. These methods do 2 things: (1) allocated SQLite memory is locked, so that the OS doesn't swap it to disk, and (2) freed SQLite memory is overwritten with zeros.

sqleet defines SQLITE_TEMP_STORE=2 to avoid storing temporary files on disk. Obvious sensitive information such as derived encryption keys are overwritten with zeros when no longer used.
SQLCipher's more thorough approach is common in security industry, but I personally do not think that the security-performance trade-off is worth it in the context of SQLite3. If the attacker has access to the swap file, (s)he can usually also read the SQLite3 process memory directly in one way or another. And besides encryption keys & decrypted database pages, there should not be anything security-critical information laying around in unused heap memory (however, I'm not an expert when it comes to SQLite3's internal memory management so I might be very wrong).
...  I just now realized that sqleet does not encrypt database pages in-place, which may leave cleartext copies of database pages in SQLite3's (heap) memory. My reasoning behind this design decision was that SQLite3 might try to erroneously access the encrypted buffer, which could crash the program. Of course this is very unlikely case (and would indicate a bug in SQLite3's code), and certainly not a good enough reason to leave unencrypted copies of database pages in the memory. I'm fixing this ASAP (although the possible information leak is miniscule and requires use-after-free or memory leak vulnerability to actually exploit).
Defining custom memory handling methods is easy though (see Memory Allocation Routines). So making SQLCipher-style memory allocation behavior optional with compile-time or run-time configuration setting would be a "Pareto-optimal" solution, which I might consider in the future. The potential benefit is insignificant in my estimation, but if some users want it, I'm not against implementing custom memory routines.

Anyway, commit 957fbb8 flags the current raw key/salt interface as experimental, which I should've done when first introducing it. I might replace the interface with something better based on our discussions if preserving it does not make sense after the changes. Currently URI query parameters feel like the best method for passing extra settings.
Implementation is coming perhaps sometime in January. Happy holiday season, don't spend it all in GitHub. :evergreen_tree:

The extra housekeeping does not necessarily require too much extra logic if implemented carefully and the basic usage would remain simple, i.e., only setting the key once. The PRAGMA ordering is indeed mistake-prone, but IMO the order requirements are intuitive enough (the only thing that matters is setting the password after other settings).

Instead of introducing multiple use of PRAGMA key= with various prefixes, one could introduce a special syntax to specify all parameters within a single invocation of the pragma. However, even with this simplification I don't think that this approach is a good one. Especially, the error handling would be cumbersome to say the least, because the SQLite code invoking the sqlite3_key interface does not check the return code, unfortunately.

However, I'm still unsure about how rekey should work under this model.

AFAICT the original purpose of sqlite3_rekey is only to allow to change the password/passphrase of the current encryption or to remove encryption. However, allowing to set/modify the key salt is certainly desirable, too.

Maintaining UX compatibility between sqleet and wxSQLite3 would also be nice.

Yes, indeed.

The idea is to use a VFS shim to add an extra SQLite3 format 3\x00 header in the beginning of the encrypted database file. The VFS shim could then redirect SQLite3's reads & writes by an offset of +16, basically skipping the fake header. As a result, iOS would recognize the database file as a SQLite3 database, but from the perspective of SQLite3/sqleet, the file would look like a regular encrypted database when accessed through the VFS shim.

Just adding an extra SQLite3 format 3\x00 header would not be enough, since at least the next 8 bytes are also inspected by iOS. And it would be necessary to keep these 8 bytes in sync with the actual database header.

The 16-byte offset may incur a noticeable performance cost though. Extending the fake header to database page boundary might help in that regard (or not).
But yeah, this feels like a dirty iOS-specific hack and not a proper solution.

Definitely.

sqleet defines SQLITE_TEMP_STORE=2 to avoid storing temporary files on disk.

This is the default for wxSQLite3 as well.

SQLCipher's more thorough approach is common in security industry, but I personally do not think that the security-performance trade-off is worth it in the context of SQLite3. If the attacker has access to the swap file, (s)he can usually also read the SQLite3 process memory directly in one way or another. And besides encryption keys & decrypted database pages, there should not be anything security-critical information laying around in unused heap memory (however, I'm not an expert when it comes to SQLite3's internal memory management so I might be very wrong).

Well, SQLite caches decrypted database pages in memory. That is, if these cached pages are written to a swap file, an attacker with access to the swap file could read the unencrypted database content at least partially.
However, even SQLCipher's approach isn't foolproof, since OSs usually limit the amount of memory that can be locked.

... I just now realized that sqleet does not encrypt database pages in-place, which may leave cleartext copies of database pages in SQLite3's (heap) memory.

IMHO this doesn't seem to add a major zone for attack, since SQLite caches cleartext database pages anyway.

Apparently in-place encryption is not trivial and as straightforward as it should be.

I really don't think it's worth the effort to implement in-place encryption, unless one prevents caching cleartext database pages as well. However, the latter would incur a major performance decrease.

Defining custom memory handling methods is easy though (see Memory Allocation Routines). So making SQLCipher-style memory allocation behavior optional with compile-time or run-time configuration setting would be a "Pareto-optimal" solution, which I might consider in the future. The potential benefit is insignificant in my estimation, but if some users want it, I'm not against implementing custom memory routines.

Well, yes, adding custom memory handling seems to be rather straight-forward. However, it might make sense to ask the user base whether there is an actual demand for such a feature.

Anyway, commit 957fbb8 flags the current raw key/salt interface as experimental, which I should've done when first introducing it.

Yeah, I should have done that, too, with (at least some of) the changes I introduced with version 4 of wxSQLite3.

Currently URI query parameters feel like the best method for passing extra settings.

In any case using URI query parameters is an established method to pass configuration settings to VFS implementations or the like. So, using this mechanism for encryption configuration is definitely reasonable.

Implementation is coming perhaps sometime in January. Happy holiday season, don't spend it all in GitHub. :evergreen_tree:

I followed your advice. :smile:

Oops, sorry for the reference spam.
Anyway, this took longer than expected. The implemented URI API in uri branch is experimental. It is reasonably tested and in a usable state, but some feedback and minor adjustments are needed before merging it to master (README.md documentation is also still missing). The commit message of 2de2f50 gives a good overview of the chosen approach.
Implement URI API for run-time configuration (#13)

Supported URI query parameters by sqleet:

  * salt: Provides 16-byte salt for the key derivation function (KDF)
  * header: Overrides 16-bytes in the beginning of the database header
  * kdf: Specifies KDF algorithm (only `none` value supported for now)
  * skip: Run-time setting overriding compile-time SKIP_HEADER_BYTES
  * page_size: Equivalent to `page_size` PRAGMA setting

Parameters `salt` and `header` have corresponding hex-prefixed versions.
Both expect 16-byte values and shorter strings are 0-padded to 16 bytes.
The KDF salt is stored in the first 16 bytes of the database header if
`header` is undefined. Otherwise the value of `header` overwrites the 16
header bytes (the KDF salt can thus be hidden from the database file).

Old raw key interface is deprecated by `kdf=none` that disables the KDF.
When KDF is disabled, `key` and `rekey` PRAGMAs (and corresponding URI
parameters) accept a 32-byte value that becomes the "master" key which
is normally derived by the KDF. This allows the users of the library to
take full control of the key derivation process if needed.

There is not yet a user-friendly method to retrieve the currently used
salt or other settings. Moreover, changing the settings of an existing
database after its creation remains cumbersome. It can however be done
by first fully decrypting the database (with `PRAGMA rekey=''`) and then
encrypting it again (with `PRAGMA rekey='password'`) after re-opening
the database file with new URI parameters. Addressing these shortcomings
is a low-priority long-term goal given the workarounds and the lack of
low-complexity solutions to them without a major code refactoring.

README.md documentation of the URI API is left for a later commit.

I agree that this is a bit different and less powerful than existing solutions in SQLCipher/wxSQLite3, but I believe it is adequate for most common use-cases. For example, iOS applications can use sqleet as follows:
[sqleet]% ./sqleet 'file:secrets.db?key=swordfish&salt=SodiumChloride42&header=SQLite%20format%203&skip=32'
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> CREATE TABLE f(x,y);
sqlite> .quit
[sqleet]% xxd secrets.db | head -n5
00000000: 5351 4c69 7465 2066 6f72 6d61 7420 3300  SQLite format 3.
00000010: 1000 0101 2040 2020 0000 0001 0000 0002  .... @  ........
00000020: 9d96 552c 3b57 e8fd 70e6 4631 a6c7 b0ce  ..U,;W..p.F1....
00000030: 0f9c f4f6 ef41 bc9e 926d f55e 7d7d 8835  .....A...m.^}}.5
00000040: d01b 5d08 a9fb 1c62 3db0 e814 6948 088c  ..]....b=...iH..

For illustration purposes, the same without header:
[def@arch sqleet]% ./sqleet 'file:secrets.db?key=swordfish&salt=SodiumChloride42&skip=32'
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> CREATE TABLE f(x,y);
sqlite> .quit
[def@arch sqleet]% xxd secrets.db| head -n5
00000000: 536f 6469 756d 4368 6c6f 7269 6465 3432  SodiumChloride42
00000010: 1000 0101 2040 2020 0000 0001 0000 0002  .... @  ........
00000020: cd80 04f7 33dc 482c 75a0 b3d6 7c7c 1662  ....3.H,u...||.b
00000030: d7fc 2267 c33b 78bf 18cf 60fe e9da e27a  .."g.;x...`....z
00000040: b5b2 b355 4c96 9428 19b8 cb6d f3c8 4c12  ...UL..(...m..L.

Accessing the active salt is still unsupported by sqleet (the obvious workaround is to direcrtly read the 16-byte header of the database file). Changing the URI configuration of an existing database is also practically an unsupported operation, although it is possible with enough effort as described in the commit message. Ideally, sqleet would make these things easier. However, I'm uncertain about the cost-benefit ratio because I'd like to keep sqleet code as simple and auditable as possible (offering read access to the salt would be relatively easy, but updating the salt and other database configuration "on-the-fly" would be very tricky).
The code base is becoming increasingly complex. It is not yet too bad, but I'll try to do some refactoring after merging the URI API and before implementing more new features.

Documentation and refactoring of the URI interface is still in progress. I have been busy lately.

I just realized that the new VACUUM INTO command can be used to change URI settings of an existing database fairly easily. The filename argument to INTO can be a URI filename with a new set of URI parameters. This is just an untested idea and most likely requires small adjustments to the current code (e.g., the logic that duplicates the main database's codec when attaching a database without a key should overwrite duplicated URI settings with newly parsed values).

Commit 3fc3fc4 in uri branch adds README.md documentation of the URI interface.
Some work is still needed before merging the feature into master. In particular, support for changing URI settings with VACUUM INTO  should be implemented and tested properly.

Based on initial testing, the VACUUM INTO thing seems to work perfectly for changing URI settings of an existing database.

Current uribranch now supports changing database URI settings via VACUUM INTO command.
$ rm -f secrets.db skipped.db
$ ./sqleet 'file:secrets.db?key=hunter2&salt=SALTsaltSALTsalt' 'CREATE TABLE f(x,y)'
$ xxd secrets.db | head -n3
00000000: 5341 4c54 7361 6c74 5341 4c54 7361 6c74  SALTsaltSALTsalt
00000010: dc01 5e20 8fc5 e8ae 52c3 83fb 44b0 1550  ..^ ....R...D..P
00000020: ea0b 8237 94b4 d99f 74af e80b 7bbd c537  ...7....t...{..7
$ ./sqleet 'file:secrets.db?key=hunter2' 'VACUUM INTO "file:skipped.db?skip=32"'
$ xxd skipped.db | head -n3
00000000: 5341 4c54 7361 6c74 5341 4c54 7361 6c74  SALTsaltSALTsalt
00000010: 1000 0101 2040 2020 0000 0001 0000 0002  .... @  ........
00000020: 93f7 e5f7 e31e bc9c ae57 7f48 39fc 8498  .........W.H9...

Salt is stored in the DB header so the salt parameter can be omitted in the latter command. Source database values of URI parameters salt, header, page_size, skip and kdf are preserved unless explicitly overridden in the INTO URI filename
URI parameter key allows (re-)encrypting & decrypting databases. Unfortunately, changing page_size of an existing database is unsupported: VACUUM INTO calls sqlite3RunVacuum() which overwrites page size and reserved size without our control. This is basically the same issue that occurs in rekeying, but in this case we cannot solve it with sqlite3RekeyVacuum() since VACUUM INTO calls sqlite3RunVacuum() directly. However, this is only a minor limitation and most people can probably live with fixed database page size (or change it by various other means).

URI-based configuration is finally getting pretty close to the finish line. No further code refactoring is needed in my opinion. Only README.md needs some updating.

@utelle we are facing this very problem while using SQLiteMultipleCiphers. I know this project is discontinued but were you able to or think of adding support for plaintext header like SQLCipher in SQLiteMultipleCiphers?

@utelle we are facing this very problem while using SQLiteMultipleCiphers. I know this project is discontinued but were you able to or think of adding support for plaintext header like SQLCipher in SQLiteMultipleCiphers?

Up to now no one has asked for a "plaintext header" feature in SQLite3 Multiple Ciphers for other cipher schemes than SQLCipher. That is, this feature is currently only implemented for the SQLCipher cipher scheme.
If you need this feature for other cipher schemes, please open a SQLite3 Multiple Ciphers issue, and I will see if and when it can be added.

authors: resilar thecodrr utelle

repo: resilar/sqleet

license: unlicense


text:
python instructions

Hello and thanks for this awesome project.
I just wanted to know how can i use this library as the default sqlite library in my python projects?
i am naive developer i was curios if i should download the source of sqlite3 amalgamation and provide c headers of this project to compile with. am i right?
BTW what should i do after compiling sqlite3 for integrating it into my python projects?
thanks in advance.

I am not aware of an easy way to use sqleet in Python projects. However, you are not completely out of luck thanks to pysqlite3 by Charles Leifer (see this blog post for his instructions). You can compile sqleet for use in Python virtualenv as follows:
% virtualenv -p /usr/bin/python3 sqleet
% cd sqleet
% source bin/activate
% git clone https://github.com/coleifer/pysqlite3
% cd pysqlite3
% wget https://github.com/resilar/sqleet/releases/download/v0.27.1/sqleet-v0.27.1-amalgamation.tar.gz
% tar -xvf sqleet-v0.27.1-amalgamation.tar.gz
% cp sqleet-v0.27.1/sqleet.c sqlite3.c
% cp sqleet-v0.27.1/sqleet.h sqlite3.h
% python setup.py build_static
% python setup.py install
% python3
Python 3.7.2 (default, Jan 10 2019, 23:51:51)
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pysqlite3 import dbapi2 as sqleet
>>> sqleet.sqlite_version
'3.27.1'

The process is actually rather simple thanks to the standalone nature of sqleet (there is no need to configure OpenSSL dependencies or other stuff). Usage on Windows/Mac probably requires a bit more effort and I cannot help with that at the moment.

Ultimately, we would like to have pysqleet Python package that can be installed with PIP for any operating system. This, however, is a project for someone else than me.
Leave this issue open until we have proper Python support. It can take a long time though.

Well awesome! your kind answer was really helpful to me.
Thanks in advance.

I created a GitHub organization for sqleet: https://github.com/sqleet. It is currently empty, but the plan is to add repos of sqleet wrappers/packages for different programming languages and target platforms (for example, Android/iOS wrappers and Python/node.js packages). This main sqleet repo should also be transferred to that organization at some point in the future.
Obligatory disclaimer: Hold your horses! This is a long-term plan for upcoming years, so do not expect specialized support for your favorite language anytime soon. Unless, of course, you are willing to implement that support yourself. Contact me if you want your wrapper/package to be added to the sqleet organization.

Apparently someone has made a pypi package for sqleet. https://pypi.org/project/apsw-sqleet/
Combining all existing sqleet wrappers/packages to one place would be nice. However, not in the scope of this issue. Closing.

Also https://pypi.org/project/pysqleet/

authors: clach04 pouyamiralayi resilar


text:
fix issue, where only one day of history shown

end_time was assumed to be one day after after startDate by HA. providing end_time=now() explicitly fixes this.
see https://community.home-assistant.io/t/tileboard-new-dashboard-for-homeassistant/57173/2815

Here's the HA API documentation:

Thanks!

authors: akloeckner resoai

repo: resoai/TileBoard


text:
[RESTEASY-1728] Look for resteasy.allowGzip system property in Regist…

…erBuiltin
This is a proposal for a "fix" applying to both client and server side. Basically every time a new ResteasyProviderFactory instance is created,we would go and check the system property.
This said, we can probably do better... please provide feedback.

This is related to the issue that got me thinking about a general purpose facility for checking system properties. Something like
         ResteasyConfiguration context = ResteasyProviderFactory.getContextData(ResteasyConfiguration.class);
         if (context != null)
         {
            String s = context.getParameter("resteasy.allowGzip");
            if (s == null)
            {
                 s = AccessController.doPrivileged(new PrivilegedAction<String>() {
                        @Override
                        public String run() {
                            return System.getProperty("resteasy.allowGzip");
                        }
            }
            if (s != null)
            {
               allowGzip = Boolean.parseBoolean(s);
            }
         }

@ronsigal Take a look at Elytron's security manager facility, https://github.com/wildfly-security/wildfly-elytron/blob/master/src/main/java/org/wildfly/security/manager/WildFlySecurityManager.java#L1114.

@ronsigal , sure, it's related, and this is probably yet another confirm that we should be implementing your proposal (in that case we'd refactor this fix a bit, should be possible in backward compatible way, right?).
@iweiss , what do you actually want to suggest with the link above?

Do you guys think this can be used in 2.3.x branch as it is, so that it's not holding EAP release any longer? If there's going to be some refactoring later on, this branch could perhaps live without it, if it doesn't change a substance of the fix.

authors: TomasHofman asoldano iweiss ronsigal

repo: resteasy/Resteasy


text:
RESTEASY-2739 - Update WADL section document

Updated part:

Related PR:

https://github.com/resteasy/resteasy-examples/pull/61
https://github.com/resteasy/Resteasy/pull/2568


text:
Diagnostic command to display what snapshots a blob/tree belongs to

This is not a bug report but a feature proposal.
A few times, I've been missing a command to list the snapshots that contain a specific tree or blob, or the snapshots affected by a corrupted pack.
I'd like to be able to use a restic command for this case (AFAIK it does not exist yet) and I've written some POC code.
I can do a PR to discuss this further if this suggestion sounds reasonable to you.
Here's a practical example.
% restic check
repository a04959d0 opened successfully, password is correct
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
(...)
error for tree e437ff75:
  tree e437ff75: file "checks.db": metadata size (122171392) and sum of blob sizes (122175488) do not match
Fatal: repository contains errors

Let's check what this tree actually is and which snapshot is affected.
% restic what-contains tree e437ff75
repository a04959d0 opened successfully, password is correct
Found tree e437ff75
 ... path /mikael/check-data
 ... in snapshot f120bfbf

Here's another example with a random blob in one of my repositories:
% restic what-contains blob 9dcf5f6a254fb58c8b2a7e604dc76b71fbad847f6ade751513968cf2f1f161fc
repository a04959d0 opened successfully, password is correct
Blob found in pack cf8a3fa3b2c9c7a08b6b6fc6cce9fdc2eeb4dd26ebd2d67b081d63f24dd408d8
 ... Pack cf8a3fa3: <Blob (data) 9dcf5f6a, offset 2486760, length 12214>
Found blob 9dcf5f6a254fb58c8b2a7e604dc76b71fbad847f6ade751513968cf2f1f161fc
 ... in file 1398431273.H215567P22415.mail.example.com:2,
 ... in snapshot 067f149f
Found blob 9dcf5f6a254fb58c8b2a7e604dc76b71fbad847f6ade751513968cf2f1f161fc
 ... in file 1398431273.H215567P22415.mail.example.com:2,
 ... in snapshot 0a781e61
Found blob 9dcf5f6a254fb58c8b2a7e604dc76b71fbad847f6ade751513968cf2f1f161fc
 ... in file 1398431273.H215567P22415.mail.example.com:2,
 ... in snapshot 15ab2468
(...)

I'd also like to be able to do
% restic what-contains pack cf8a3fa3

but I haven't had a look at it yet.
Please let me know if it would be useful to other people than me.

I have added support for packs (see the PR):
restic what-contains pack 0014cae1664cda85b69d22c3e0d86fe56085aebc3333613b7a956ea854dab118
repository a04959d0 opened successfully, password is correct
Looking for packs...
Found pack 0014cae1664cda85b69d22c3e0d86fe56085aebc3333613b7a956ea854dab118
33 blobs found
Found blob 1c4925a45401e8ecf71c2dc1a0294a823419839b122c80c973c7fc95d0a8a87b
 ... in file ____init____.py.i
 ... in snapshot 00ccb2af
Found blob 909975ec21ccf6b28c05a88a7281a0b3fd7ee235d31eec73eced46ef2efd9835
 ... in file posix.py.i
 ... in snapshot 00ccb2af
Found blob 46dbe70ab31e84c542bb0b226cd3bce0d33bc0807b56ab37f434eae32cc07692
 ... in file ch02_19.htm
 ... in snapshot 5b7eca88
(...)

This looks great, thank you so much for spending time on it! I'm currently swamped with other real-life issues, and I've got a huge backlog of things to review. Sorry for that!

Closing this issue as the Merge Request #1780 has been accepted.

repo: restic/restic


text:
fuse mount losing sticky bit

# mkdir r; dd if=/dev/random bs=32 count=1 of=p status=none; ./restic-0.16.0 version; ./restic-0.16.0 -q init -r r -p p; ./restic-0.16.0 -q -r r -p p backup /tmp; ./restic-0.16.0 -q -r r -p p mount /mnt
restic 0.16.0 compiled with go1.20.6 on linux/amd64
Now serving the repository at /mnt
Use another terminal or tool to browse the contents of this folder.
When finished, quit with Ctrl-c here or umount the mountpoint.

# rm -rf foo; cp -a /mnt/snapshots/latest/tmp foo; ls -lahd /tmp /mnt/snapshots/latest/tmp foo
drwxrwxrwx 14 root root 4,0K Sep 27 10:18 foo
drwxrwxrwx  2 root root    0 Sep 27 10:18 /mnt/snapshots/latest/tmp
drwxrwxrwt 14 root root 4,0K Sep 27 10:18 /tmp

I expected the sticky bit to be present on the /tmp directory in the fuse mount, but it wasn't.

Clearly a defect in restic mount. The tarball from restic dump does have the sticky bit on the directory.

I would like to help with this issue
@greatroar can you tell me how did you check with restic dump ? I don't see the sticky bit even with dump command
batman@fedora:~/fun/restic$ ./restic -r ../test dump latest tmp > dumpp.tar
enter password for repository:
batman@fedora:~/fun/restic$ ls -lahd dumpp.tar
-rw-r--r--. 1 batman batman 247K Dec 24 01:34 dumpp.tar
batman@fedora:~/fun/restic$ rm -rf tmp/
batman@fedora:~/fun/restic$ tar -xf dumpp.tar
batman@fedora:~/fun/restic$ ls
build.go      cmd              doc     dumpp      go.mod         helpers   Makefile   restic  VERSION
changelog     contrib          doc.go  dumpp.tar  go.sum         internal  README.md  steps
CHANGELOG.md  CONTRIBUTING.md  docker  foo        GOVERNANCE.md  LICENSE   res        tmp
batman@fedora:~/fun/restic$ ls -lahd tmp/
drwxr-xr-x. 1 batman batman 1.1K Dec 24 01:35 tmp/
batman@fedora:~/fun/restic$

I did RCA of this issue

This issue existed because in anacrolix/fuse@v0.2.0, sticky bit was silently just being dropped (SUID and SGID were maintained just fine). This issue was fixed in the commit - https://github.com/anacrolix/fuse/commit/d711f1e6391d975060229ebcd53168d6c012e74a
Restic is currently using v2 of this fuse library but the fix above is there in v3
I tried upgrading to v3 but it seems that it had issues working on linux (NOTE: The latest master of fuse works fine - https://github.com/anacrolix/fuse/commit/c6729d8bb2af377b5df04d4efdc425526cb764f3)
With the latest master, I can see the sticky bit persisting in the mounted directory

I've taken a closer look at the changes in v0.3 of the fuse library and found a few problems that are fixed in https://github.com/anacrolix/fuse/pull/6. These changes untested so far, as I don't have a mac anymore. Once the bugfixes (in particular the crash fix) have been merged, we could just go forward and bump the dependency.

Restic 0.17.0 will include version 0.3.1 of anacrolix/fuse, which includes the bugfix for the sticky bit.

authors: MatthiasFoerste MichaelEischer gautammenghani greatroar


text:
Document linktarget_raw field for non utf symlinks

What does this PR change? What problem does it solve?
Follow up to #3802

Was the change previously discussed in an issue or on the forum?
Fixes https://github.com/restic/restic/pull/3802#issuecomment-1646748582

Checklist

[x] I have read the contribution guidelines.
[x] I have enabled maintainer edits.
[ ] I have added tests for all code changes.
[x] I have added documentation for relevant changes (in the manual).
[ ] There's a new file in changelog/unreleased/ that describes the changes for our users (see template).
[x] I have run gofmt on the code in all commits.
[x] All commit messages are formatted in the same style as the other commits in the repo.
[x] I'm done! This pull request is ready for review.

LGTM.


text:
Updates for build.go

Hi Alexander,
Some small changes to build.go.

Nice work, thanks for your contribution!


text:
Dashboard: read only on the editor

Hello 👋,
👀 Background
The RestQA Dashboard could be deployed on a remote server. However some team would like to deploy the server in order to access to all the feature and run any of them.
✌ What is the actual behavior?
Each feature file could be updated by anyone through the dashboard UI.
🕵♀ How to reproduce the current behavior?

Install RestQA
Run the command restqa init to initialize a brand new project
Run the command restqa dashboard to launch the dashboard
Access the dashboard through the url http://localhost:8081
Go to Editor menu and update one feature file.

🤞 What is the expected behavior?
On some team setup we would prefer to not allow the user to edit the the feature when the restqa dashboard is deployed on a remote server.
😎 Proposed solution.
On the .restqa.yml configuration file , we could add the property editable is the object restqa.dashboard such as :
restqa:
  dashboard:
    editable: true

The default value will be true.
The value could also be overrode by the environment variable RESTQA_DASHBOARD_EDITABLE=true, example:
RESTQA_DASHBOARD_EDITABLE=false restqa dashboard

Or through the  option -e | --editable
restqa dashboard --editable false

Cheers.

Hi @olivierodo
As you see, I pushed few lines on the PR #120. From the feature POV, it just misses to handle the --editable arg.
But, it remains a question mark on my side: regarding the environment variables. As we have a configuration file, Which one is the real source of truth ?
I mean what's happen if the dashboard environement variables say "editable" and config files say "not editable" ?
If we want to move forward quickly, we can open a specific issue for that continue to work on the feature without this "env var" aspect.
Let me know.

Owww maybe the env variable you mention is the one that will be used by the CLI to generate the restqa config file ?

No actually from a team context perspective we could face the following pattern:

Dev team run the dashboard on local so the flag  of restqa.dashboard.editable will be set to true.
Dev team are committing the config file as-is.
The continuous integration pipeline will deploy the updated dashboard into a server
However the dashboard hosted on the server  should not be editable
Then we could imagine when the devops configure restqa on the server he could use: RESTQA_DASHBOARD_EDITABLE=false restqa dashboard in order to override the configuration set from the .restqa.yml

Is the context helping you more to understand this specific part ?

In order to enhance the user experience and the reuse common terms I think we should switch the term editable by read-only.

In the configuration file:

restqa:
  dashboard:
    read-only: true

Environement variable

RESTQA_DASHBOARD_READONLY=true  restqa dashboard

Command line options

restqa dashboard --read-only

authors: olivierodo tony-go

repo: restqa/restqa


text:
🚀 [FEATURE]: Change Feature name of generated scenario

👀 Background
During the first run, RestQA is initializing the project.
✌ What is the actual behavior?
During the initialization RestQA will generate a Test scenario of curl localhost:{port}
🕵♀ How to reproduce the current behavior?

go in the folder example/technologies/express
delete the folder tests
Delete the file .restqa.yml
run the command npm test

🤞 What is the expected behavior?
Currently the file tests/local/get.feature is generated with the

Feature name : Welcome to the RestQA community
Scenario name: Initial scenario

We should propose a standard format.
😎 Proposed solution.
Let's use the following:

Feature name : GET /
Scenario name: Default endpoint

🙏 Would you be willing to submit a PR?
Yes
Code of Conduct

[X] I agree to follow this project's Code of Conduct

File to update: https://github.com/restqa/restqa/blob/c94728a54175b3a87f4926b7fdd6812a057a1026/packages/cli/src/core/initialize.js#L150-L155


text:
Z5: Export does not show in Automatic export

Hi,
I have problem with showing export in the Automatic export. I export as BetterBibTex with selected Notes and Keep updated. But none of the exports is shown in Automatic export panel. I have tried with different Collections, but it does not show up. I am sending two errors.
Error ID: CMEELQRZ
Error ID: NSNCUKYY
Keep the good work!

Auto export is not yet implemented.

authors: mijalche retorquere

repo: retorquere/zotero-better-bibtex


text:
Fbank

Wenn man geld in die F Bank einzahlt zeigt es sich immernoch bei bargeld an

Hallo,
um Dir bestmöglich zu helfen, benötigen wir mehr Informationen. Dazu zählen zum Beispiel Deine verwendete Minecraft Version, Addon Version und Minecraft Log oder Absturzbericht.
Halte Dich bitte an die von uns bereitgestellte Vorlage. Dadurch kannst Du keine relevanten Informationen vergessen.
Für schnelle Hilfe kannst Du auch gern auf unseren Discord joinen.

Aufgrund fehlender weiterer Informationen und keiner gefundener Fehler, kann dieses Issue nicht weiter verfolgt werden.