text:
Fedora Linux: Flatpak version isn't importing or listing installed addons, AppImage spins endlessly

Hello there!
I installed first via Flatpak, gave permissions to the addons and Minion folders, but it never shows my addons nor it imports from Minion, which I'm not sure to what file to point to ($HOME/.minion/minion.xml?).
Trying the AppImage, it opens to a spinning state and never ends.
I'm on Fedora 39, both minion manual and from Flatpak works with the same folders. Any other permissions to give?
[flatpak version]

[AppImage]

Both AppImage and the "unknown Linux gnu" version show this when ran from a terminal:
 ./eso-addon-manager-x86_64-unknown-linux-gnu
2024-01-30T14:28:28.917351Z  INFO winit::platform_impl::platform::x11::window: Guessed window scale factor: 1
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: ConfigParse { source: Error("EOF while parsing a value", line: 1, column: 0), path: "/home/marcelo/.config/eso-addons/config.json" }', core/src/config.rs:86:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/main.rs:102:37
2024-01-30T14:28:34.405397Z  WARN egui_glow::painter: You forgot to call destroy() on the egui glow painter. Resources will leak!

And both keep spinning endlessly.

❯ cargo run
warning: unused import: `Path`
 --> core/src/service/mod.rs:4:17
  |
4 | use std::path::{Path, PathBuf};
  |                 ^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: unused variable: `update`
   --> core/src/service/mod.rs:185:13
    |
185 |         for update in updates.iter() {
    |             ^^^^^^ help: if this is intentional, prefix it with an underscore: `_update`
    |
    = note: `#[warn(unused_variables)]` on by default

warning: method `get_missing_addon_detail_ids` is never used
   --> core/src/service/mod.rs:291:14
    |
47  | impl AddonService {
    | ----------------- method in this implementation
...
291 |     async fn get_missing_addon_detail_ids(&self) -> Result<Vec<i32>> {
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(dead_code)]` on by default

warning: function `fs_get_addon` is never used
  --> core/src/service/fs_util.rs:22:8
   |
22 | pub fn fs_get_addon(addon_dir: &PathBuf, name: &str) -> Result<Option<...
   |        ^^^^^^^^^^^^

warning: function `fs_get_addons` is never used
  --> core/src/service/fs_util.rs:28:8
   |
28 | pub fn fs_get_addons(addon_dir: &PathBuf) -> Result<AddonList> {
   |        ^^^^^^^^^^^^^

warning: `eso-addons-core` (lib) generated 5 warnings (run `cargo fix --lib -p eso-addons-core` to apply 2 suggestions)
warning: unused import: `VecDeque`
 --> src/views/installed.rs:1:33
  |
1 | use std::collections::{HashMap, VecDeque};
  |                                 ^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: unused variable: `ctx`
   --> src/views/addon_details.rs:109:9
    |
109 |         ctx: &egui::Context,
    |         ^^^ help: if this is intentional, prefix it with an underscore: `_ctx`
    |
    = note: `#[warn(unused_variables)]` on by default

warning: unused variable: `addon_id`
  --> src/views/browse.rs:41:29
   |
41 |     fn install_addon(&self, addon_id: i32, service: &mut AddonService) {
   |                             ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_addon_id`

warning: unused variable: `service`
  --> src/views/browse.rs:41:44
   |
41 |     fn install_addon(&self, addon_id: i32, service: &mut AddonService) {
   |                                            ^^^^^^^ help: if this is intentional, prefix it with an underscore: `_service`

warning: unused variable: `ctx`
  --> src/views/browse.rs:49:9
   |
49 |         ctx: &eframe::egui::Context,
   |         ^^^ help: if this is intentional, prefix it with an underscore: `_ctx`

warning: unused variable: `service`
  --> src/views/missing_deps.rs:36:24
   |
36 |     fn poll(&mut self, service: &mut AddonService) {
   |                        ^^^^^^^ help: if this is intentional, prefix it with an underscore: `_service`

warning: unused variable: `ctx`
   --> src/views/missing_deps.rs:100:9
    |
100 |         ctx: &eframe::egui::Context,
    |         ^^^ help: if this is intentional, prefix it with an underscore: `_ctx`

warning: unused variable: `i`
   --> src/views/ui_helpers.rs:288:53
    |
288 | ...egui::Ui, node: &BBNode, i: i32, parent_nodes: &[BBNode]) {
    |                             ^ help: if this is intentional, prefix it with an underscore: `_i`

warning: associated function `new` is never used
  --> src/views/ui_helpers.rs:61:12
   |
60 | impl<T: Send + Clone + Default> PromisedValue<T> {
   | ------------------------------------------------ associated function in this implementation
61 |     pub fn new(value_promise: ImmediateValuePromise<T>) -> Self {
   |            ^^^
   |
   = note: `#[warn(dead_code)]` on by default

warning: `eso-addon-manager` (bin "eso-addon-manager") generated 9 warnings (run `cargo fix --bin "eso-addon-manager"` to apply 8 suggestions)
    Finished dev [unoptimized + debuginfo] target(s) in 1.05s
     Running `target/debug/eso-addon-manager`
2024-01-30T14:44:12.043592Z  INFO winit::platform_impl::platform::x11::window: Guessed window scale factor: 1
2024-01-30T14:44:12.346344Z  INFO eso_addons_core::config: Loading config data at: /home/marcelo/.config/eso-addons/config.json
2024-01-30T14:44:12.347133Z  INFO eso_addons_core::api: Requesting: https://api.mmoui.com/v3/globalconfig.json
2024-01-30T14:44:12.578263Z  INFO eso_addons_core::api: Requesting: https://api.mmoui.com/v3/game/ESO/gameconfig.json
2024-01-30T14:44:12.756311Z  INFO sea_orm_migration::migrator: Applying all pending migrations
2024-01-30T14:44:12.757508Z  INFO sea_orm_migration::migrator: No pending migrations
2024-01-30T14:44:12.779134Z  INFO eso_addon_manager::views::installed: Checking for updates
2024-01-30T14:44:12.779270Z  INFO eso_addons_core::service: Updating categories
2024-01-30T14:44:12.779296Z  INFO eso_addons_core::api: Requesting: https://api.mmoui.com/v3/game/ESO/categorylist.json
2024-01-30T14:44:12.957845Z  INFO eso_addons_core::api: Requesting: https://api.mmoui.com/v3/game/ESO/filelist.json
2024-01-30T14:44:13.774524Z  INFO eso_addons_core::service: Checking for missing dependencies
2024-01-30T14:44:13.775339Z  INFO eso_addons_core::service: Saving config
2024-01-30T14:44:13.779716Z  INFO eso_addon_manager::views::installed: Getting installed addons
2024-01-30T14:44:13.779848Z  INFO eso_addons_core::service: Getting installed addons
2024-01-30T14:44:13.781729Z  INFO eso_addons_core::service: Done getting addons!
2024-01-30T14:44:13.792863Z  INFO eso_addon_manager::views::installed: Sorting addons
thread 'tokio-runtime-worker' panicked at core/src/service/mod.rs:742:43:
called `Result::unwrap()` on an `Err` value: ParseIntError { kind: InvalidDigit }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2024-01-30T14:45:36.663051Z  INFO eso_addon_manager::views::installed: Getting installed addons
2024-01-30T14:45:36.663201Z  INFO eso_addons_core::service: Getting installed addons
2024-01-30T14:45:36.664790Z  INFO eso_addons_core::service: Done getting addons!
2024-01-30T14:45:36.676529Z  INFO eso_addon_manager::views::installed: Sorting addons
2024-01-30T14:45:41.757197Z  INFO eso_addon_manager::views::installed: Getting installed addons
2024-01-30T14:45:41.757321Z  INFO eso_addons_core::service: Getting installed addons
2024-01-30T14:45:41.759178Z  INFO eso_addons_core::service: Done getting addons!
2024-01-30T14:45:41.771230Z  INFO eso_addon_manager::views::installed: Sorting addons

Hmm, unfortunately I have done a poor job documenting the Minion import process. I wasn't sure how much demand there would be for this feature before releasing to flathub.
It is designed to load the BU-addons.txt file. I'm not sure where this file is located but it's possible it is generated in backup functionality of minion. It should be a plaintext file with comma-separated numbers (7,56,288,[... etc]).
If you can find that file, I would recommend trying to import it again. It should work regardless of the install method and shouldn't need additional filesystem permissions, as it requests host access.
I will work on better documenting that process in the app. I would really appreciate if you could pass on what you do to load it if you get it working (where the txt file is stored, generated, anything really), this would greatly help me in documenting the instructions.
Also, thank you for the interest in this project!

Both AppImage and the "unknown Linux gnu" version show this when ran from a terminal:
 ./eso-addon-manager-x86_64-unknown-linux-gnu
2024-01-30T14:28:28.917351Z  INFO winit::platform_impl::platform::x11::window: Guessed window scale factor: 1
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: ConfigParse { source: Error("EOF while parsing a value", line: 1, column: 0), path: "/home/marcelo/.config/eso-addons/config.json" }', core/src/config.rs:86:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/main.rs:102:37
2024-01-30T14:28:34.405397Z  WARN egui_glow::painter: You forgot to call destroy() on the egui glow painter. Resources will leak!

And both keep spinning endlessly.

Sorry about this, the appimage and linux binary in the releases page are out of date and may not work on new installs.

Hmm, unfortunately I have done a poor job documenting the Minion import process. I wasn't sure how much demand there would be for this feature before releasing to flathub.
It is designed to load the BU-addons.txt file. I'm not sure where this file is located but it's possible it is generated in backup functionality of minion. It should be a plaintext file with comma-separated numbers (7,56,288,[... etc]).
If you can find that file, I would recommend trying to import it again. It should work regardless of the install method and shouldn't need additional filesystem permissions, as it requests host access.
I will work on better documenting that process in the app. I would really appreciate if you could pass on what you do to load it if you get it working (where the txt file is stored, generated, anything really), this would greatly help me in documenting the instructions.
Also, thank you for the interest in this project!

I'll try, but even cloning and running the project locally I couldn't make it work pointing to the correct folder with the needed permissions. I'll try importing the CSV file, it's probably inside $HOME/.minion. I'm not sure about Minion's flatpak version if it's inside $HOME/.var/app/gg.minion.Minion or if it uses the same one.
I was already for a long time concerned with Minion not updating in so long. And ESO-Addons having Windows and Mac binaries, is easier to recommend to the ESO community.

A file with the name BU-addons.txt does exist on my system, I did not find it at least for both the flatpak and the java version of Minion (both pointing to the same $HOME/.minion folder. Flatpak backups I have no idea where they are stored. Couldn't find them despite they being listed inside Minion.
The backup option creates a .zip file containing a "mirror" of the path for the ESO addons folder with all sub-folders, like: mnt/Games/SteamLibrary/steamapps/compadata/306130/pfx/drive_c/users/steamuser/Documents/Elder Scrolls Online/live.AddOns. And also creates a text file containing, what I guess being the addons codes separated by commas.
Importing the .txt file listed all my addons, but the importing process never ends spinning and showing Importing... in the button. Looking at the process, it makes me thing that setting the Addons folder path is buggy since it nothing will happen until a .txt file is imported.
I think some changes, graphically at least, to make it easier, simpler and friendlier would be nice like.

Bigger tab/buttons;
Bigger font;
First run screen for helping importing/configuring;
More contrast on the lines/table to distinguish where one addon begins and other ends;
a filter inside the Browse tab;
remove the host access on the flatpak version; For flatpak it should have as minimum access to the system as possible;

It also doesn't occupy the whole window

After configuring and importing, if I close it opens like this:

all dependencies are fulfilled already, since those are, at least should be, the addons I'm using.

All of those are great points and some are in progress (onboarding, filtering). The dependency page needs cleanup and more explanation.
I agree on the host access, but it would go against my main use case which is to make easy to install and use on the Steam Deck. To manage directory access would mean switching to desktop mode and probably connecting a keyboard to enter the commands. The primary goal is to:

Install from flathub
Select game install dir
Start installing addons

But I realize that is conflicting to it's use as a general production-ready desktop app, rather than a hobby project.
I am open to PRs if someone wants to take a stab at improving the look and theme, as it is using the default egui theme. It's on my todo, but quite further down.

v0.4.8 is out on flathub now with a bunch of layout changes. Minion import should work better now.
I'm not sure if the design is any better, but most of the button sizes and headings are larger.
Also removed host fs permission, so you might want to check your addon path before changing addons.

v0.4.8 is out on flathub now with a bunch of layout changes. Minion import should work better now.
I'm not sure if the design is any better, but most of the button sizes and headings are larger.
Also removed host fs permission, so you might want to check your addon path before changing addons.

Very good. Much more legible.
Unfortunetly maximizing the windows is not adapting the table width;
The Importing process still not clear and if I confirm by clicking open, clicking the refresh button or pressing Enter, in any stage without even choosing a file, it will show as Importing without the option to cancel;
I'd like to recomment a dialog to confirm after clicking the Quit option.

authors: arviceblot msmafra

repo: arviceblot/eso-addons

license: MIT


text:
Feature Request: Can OCMapper figure out the property/var type automatically?

Right now we're using mapFromDictionaryKey() to map a variable to the class type which works fairly well. But, as I remember in Android GSON will automatically detect the type and just parse it using that class.
Just curious - is there a way OCMapper can figure this out itself?
ie: any way OCMapper could figure out that homeTeam variable is of class type GameStatusTeam and not require the mapping below?
    var homeTeam: GameStatusTeam?

...

        mapper().mapFromDictionaryKey("homeTeam", toPropertyKey:"homeTeam", withObjectType:GameStatusTeam.self, forClass:GameStatus.self);

Hey @jpage4500 Does it not work? mapping for properties should work automatically as long as they key ion dictionary matches the property name. https://github.com/aryaxt/OCMapper/blob/master/OCMapperTests/ObjectMapperTests.m#L75
The only exception would be for arrays
ex: "homeTeams" would map to "homeTeams" as long as the class name is "homeTeams" or "homeTeam"
ex: "homeTeams" would NOT map to "homeTeams" if the class name is "Team"
For mapping array ObjectMapper finds an appropriate class based on property name (singular, plural) instead of using reflection (runtime API doesn't support it)

Hi @aryaxt - it doesn't seem to work all of the time for me. For example, after receiving your update I commented out a few mappings to test. The first one worked but others failed to parse - giving lots of errors in the log.
I did notice 1 thing that's different in my code than what I originally stated.. the server returns "home_team" and my property is called "homeTeam".
I know OCMapper automatically handles this since it's working in lots of other places for me. But, could there be an issue when OCMapper needs to do both:

convert case
determine class type

Here's what I have below (which works) - but if I comment out the mapping it fails.
    var homeTeam: GameStatusTeam?
...
        mapper().mapFromDictionaryKey("home_team", toPropertyKey:"homeTeam", withObjectType:GameStatusTeam.self, forClass:GameStatus.self);

Sorry, I should have also mentioned I have a simple mapper() function to get the singleton:
    class func mapper() -> InCodeMappingProvider {
        return ObjectMapper.sharedInstance().mappingProvider as! InCodeMappingProvider
    }

hmm are you able to write a failing test?

@interface User : NSObject

@property (nonatomic, strong) NSString *firstName;
@property (nonatomic, strong) NSString *lastName;
@property (nonatomic, strong) NSDate *dateOfBirth;
@property (nonatomic, strong) NSDate *accountCreationDate;
@property (nonatomic, strong) NSNumber *age;
@property (nonatomic, strong) Address *address;
@property (nonatomic, strong) Address *homeAddress;
@property (nonatomic, strong) Address *work;
@property (nonatomic, strong) EmailConfirmation *econfirmed;
@property (nonatomic, strong) NSArray *emailConfirmations;
@property (nonatomic, strong) NSArray *comments;
@property (nonatomic, strong) NSArray *randomKeywords;

@end

source: gharchive/issue

repo: aryaxt/OCMapper

license: mit

license_type: permissive

license_source: bigquery


text:
Fail to run RIL-seq

Dear author,
I am trying to use your package RIL-seq for my bacterial chimeric read analysis. I believe I installed correctly and ran correctly but failed in the very last step somewhere.
I load my modules SAMTools/1.2, bwa 0.7.12-r1044, PySAM/0.6, PySAM/0.6, SciPy/0.9.0, Biopython/1.65 and run the first step as "RILseq-0.57/bin/map_single_fragments.py genome.fna   -g annotation.gtf   -1 trimadapt_seq3PE2_S3_1P.fastq.gz   -2 trimadapt_seq3PE2_S3_2P.fastq.gz   -d test1   -o test2   -m 1"
The program starts to run fine for like a minute until the end that I got this error:
[bam_sort_core] merging from 3 files...
[E::hts_open] fail to open file 'test1/trimadapt_seq3PE2_S3_1P.fastq_bwa.bam'
Traceback (most recent call last):
File "RILseq-0.57/bin/map_single_fragments.py", line 153, in
status = main()
File "RILseq-0.57/bin/map_single_fragments.py", line 124, in main
samfile = pysam.Samfile(bamname)
File "csamtools.pyx", line 463, in csamtools.Samfile.cinit (pysam/csamtools.c:4758)
File "csamtools.pyx", line 496, in csamtools.Samfile._open (pysam/csamtools.c:5234)
File "csamtools.pyx", line 593, in csamtools.Samfile._open (pysam/csamtools.c:6212)
IOError: file test1/trimadapt_seq3PE2_S3_1P.fastq_bwa.bam not found
I only know very basic things about python. I tried to read your "map_single_fragments.py" script and I believe the error came from line 119 to 124 about the "bamname".
I did some searches and it appeared to me that you used a python package called "tempfile" to store the temporary file which has several versions from python2 to python3. Maybe the python version I am using is wrong and this caused the problem. Could you please let me know what do you think and what exact python version you used to develop the package?
Really appreciate your kind help!!!

I suspect that bwa didn't even run hence you didn't get the output bam
file. can you just run "bwa"and see if there is any output? If it fails,
please make sure bwa is in a $PATH directory or supply the direct link
(e.g. /xxx/yyy/bwa) using the --bwa_exec
On Sun, Mar 4, 2018 at 11:28 PM, peng7418520 notifications@github.com
wrote:

Dear author,
I am trying to use your package RIL-seq for my bacterial chimeric read
analysis. I believe I installed correctly and ran correctly but failed in
the very last step somewhere.
I load my modules SAMTools/1.2, bwa 0.7.12-r1044, PySAM/0.6, PySAM/0.6,
SciPy/0.9.0, Biopython/1.65 and run the first step as
"RILseq-0.57/bin/map_single_fragments.py genome.fna -g annotation.gtf -1
trimadapt_seq3PE2_S3_1P.fastq.gz -2 trimadapt_seq3PE2_S3_2P.fastq.gz -d
test1 -o test2 -m 1"
The program starts to run fine for like a minute until the end that I got
this error:
[bam_sort_core] merging from 3 files...
[E::hts_open] fail to open file 'test1/trimadapt_seq3PE2_S3_
1P.fastq_bwa.bam'
Traceback (most recent call last):
File "RILseq-0.57/bin/map_single_fragments.py", line 153, in
status = main()
File "RILseq-0.57/bin/map_single_fragments.py", line 124, in main
samfile = pysam.Samfile(bamname)
File "csamtools.pyx", line 463, in csamtools.Samfile.cinit
(pysam/csamtools.c:4758)
File "csamtools.pyx", line 496, in csamtools.Samfile._open
(pysam/csamtools.c:5234)
File "csamtools.pyx", line 593, in csamtools.Samfile._open
(pysam/csamtools.c:6212)
IOError: file test1/trimadapt_seq3PE2_S3_1P.fastq_bwa.bam not found
I only know very basic things about python. I tried to read your
"map_single_fragments.py" script and I believe the error came from line 119
to 124 about the "bamname".
I did some searches and it appeared to me that you used a python package
called "tempfile" to store the temporary file which has several versions
from python2 to python3. Maybe the python version I am using is wrong and
this caused the problem. Could you please let me know what do you think and
what exact python version you used to develop the package?
Really appreciate your kind help!!!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/asafpr/RILseq/issues/11, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAxQURrfTkCglPxUUwyeIvKaCKLrbJr4ks5tbFxrgaJpZM4Sbgo-
.

Thanks for the quick response. I believe bwa was working fine. I send you the complete error message file as attached. As you can see, bwa was working. But for some reason I was not getting the output file.
Also, could you please provide the version you used for pysam, numpy, scipy and biopython? I could not find them in the README file.
error_message.txt
Thanks for the help! I am very excited about your package!!!

I see, I think the problem is that    -d test1   is not an absolute path,
try replacing it with the full path and try again. I will fix it in the
code (convert to absolute path) later.
On Mon, Mar 5, 2018 at 4:26 PM, peng7418520 notifications@github.com
wrote:

Thanks for the quick response. I believe bwa was working fine. I send you
the complete error message file as attached. As you can see, bwa was
working. But for some reason I was not getting the output file.
Also, could you please provide the version you used for pysam, numpy,
scipy and biopython? I could not find them in the README file.
error_message.txt
https://github.com/asafpr/RILseq/files/1781150/error_message.txt
Thanks for the help! I am very excited about your package!!!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/asafpr/RILseq/issues/11#issuecomment-370435208, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAxQUTosftgyyXeGqhOGBvyDaQwJ1xjcks5tbUsOgaJpZM4Sbgo-
.

Hey, sorry for the late response. I have been working on it and finally got RIL-seq worked! Thanks for the help! It turned out loading the right versions of the dependent modules is the key. Also, it seems by default it is always running under your path which needs to be changed to work properly.
I have another question. I am not sure if I should open other issue or I can ask here. I am not totally clear about the output. The output column include "Start of RNA1 first read“ and ”Start of RNA1 last readRNA1“. If they mean what they literally say, then how can I know the start site and end site of RNA1? Because some of my output lines have exactly the same values for these two column which makes me confused.
Thanks for the help!

I'm glad it works now, I'll change the code accordingly.
As to your second question, we don't know where the reads end since we only
use the first 25 nucleotides for mapping. We look at the first nucleotide
of each read and see where it maps to, we then accumulate all reads mapped
to the same region (in 100 bp non-overlapping windows) and report for each
interaction where the first nt of reads span along this region. If the
value in both columns is the same it means that all reads mapped to this
region in this interaction are mapped to the same position.
On Sun, Mar 18, 2018 at 3:23 AM, peng7418520 notifications@github.com
wrote:

Hey, sorry for the late response. I have been working on it and finally
got RIL-seq worked! Thanks for the help! It turned out loading the right
versions of the dependent modules is the key. Also, it seems by default it
is always running under your path which needs to be changed to work
properly.
I have another question. I am not sure if I should open other issue or I
can ask here. I am not totally clear about the output. The output column
include "Start of RNA1 first read“ and ”Start of RNA1 last readRNA1“. If
they mean what they literally say, then how can I know the start site and
end site of RNA1? Because some of my output lines have exactly the same
values for these two column which makes me confused.
Thanks for the help!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/asafpr/RILseq/issues/11#issuecomment-373965464, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAxQUQpRF_Dh6HpYxkD4ObYviSHNE45pks5tfbb7gaJpZM4Sbgo-
.

Hey, thanks for the explanation. I am thinking it would be better, thought might be difficult, to have the full length interaction region of RNA1 and RNA2. For example, 1 to 70 bp of the read is mapped to region one and 71 to 150 bp is mapped to region two. This gives more information about where exactly the sRNAs bind to in the target RNAs.
For now, I think I can use the "Start of RNA1 first read“ info to search against the chimeric reads information obtained from map_chimeric_fragments.py and find the corresponding read name. Then I can go to the raw data to find that read and do a blast search to find the mapped two region and where they split exactly. Thanks!

I'll think about adding that. If you'll generate bed files from the
chimeric reads with the dedicated script it will give you the ranges
of each read. This is of course limited to the region being sequenced
(using paired-end) and subject to RNase digestion so the interpretation is
not straightforward.
On Sun, Mar 18, 2018 at 11:00 PM, peng7418520 notifications@github.com
wrote:

Hey, thanks for the explanation. I am thinking it would be better, thought
might be difficult, to have the full length interaction region of RNA1 and
RNA2. For example, 1 to 70 bp of the read is mapped to region one and 71 to
150 bp is mapped to region two. This gives more information about where
exactly the sRNAs bind to in the target RNAs.
For now, I think I can use the "Start of RNA1 first read“ info to search
against the chimeric reads information obtained from
map_chimeric_fragments.py and find the corresponding read name. Then I can
go to the raw data to find that read and do a blast search to find the
mapped two region and where they split exactly. Thanks!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/asafpr/RILseq/issues/11#issuecomment-374046649, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAxQUQAhs2OG0sGKZ_vmkASFF5vPxcaRks5tfsrWgaJpZM4Sbgo-
.

Thanks Asafpr! I should be able to figure out how to do this.
My research purpose is a little bit different. I don't do Hfq and RNase digestion stuff. But my sequencing result purposely generate a lot of chimeric reads. That's why I want to know the exact boundaries of the chimeras which will be combined with my normal RNA-seq result for annotation purpose. And I find RIL-seq pipelines fit my purpose well! I especially like the Fisher' exact part because my chimeric reads include a lot of background noises such as chimeric reads generated from library-prep and sequencing, and ribosomal RNAs. So, it is nice that RIL-seq can be used for other works!
Thanks again and I am closing this issue.

repo: asafpr/RILseq


text:
Which algorithm is being used for the signals on Telegram?

Could you tell me what algorithm the configuration you're running at https://t.me/intelligent_trading_signals is using? I've trained using config-sample-v0.5.0, but the results are vastly different from the signals in https://t.me/intelligent_trading_signals. Thank you for the excellent project.

The results depend not only on the algorithms but also on features defined. The Telegram bot has used different configurations for experimenting and test purposes. Currently it uses neural network and linear classifier. Previously it used gradient boosting. The algorithm parameters are defined in model_store.py in models dict.

+1

would also appreciate to get the config used shared, could be a good starting point for new comers to check against provided signals in telegram

The results depend not only on the algorithms but also on features defined. The Telegram bot has used different configurations for experimenting and test purposes. Currently it uses neural network and linear classifier. Previously it used gradient boosting. The algorithm parameters are defined in model_store.py in models dict.

Do you still use neural network and linear classifier or you changed it to something different?

Currently only NN

Why is that? NN is in your opinion the best?

I did not see big benefits from using linear classifier but it is slow to train. Actually I am using NN with only slightly more parameters than LC. For 1 hour data, I use SVM just because it is possible to train it in reasonable time. I also generate a lot of input features and in this case it is much less important which algorithm to use. It is more important to find a good trade strategy based on this (and other if available) indicators, that is, to define conditions for buying and selling. For example, how to trade in case there are intelligent indicators (scores) for 1 minute, 5 minute and 1 hour data.

feature is important

In fact, I think what the author wants to express is that you need to consider algorithm indicators in different time frames at the same time, such as issuing long signals at the same time in 5 minutes, 15 minutes, and 1 hour, and then decide to buy.

repo: asavinov/intelligent-trading-bot

license_source: github-api


text:
Addon is failing to download

I'm submitting a ...

[x ] bug report
[ ] feature request
[ ] support request

General infomration
Addon is stuck at 0% before failing
Prerequisites

[ ] Are you running the latest version?
Yes

Description
Repository is loaded and all prerequisites but the addon will not download
Steps to Reproduce

install repo from zip
click install addon

Expected behavior: [What you expected to happen]
To download and run addon
Actual behavior: [What actually happened]
Addon download freezes and is stuck at )5 before giving a fail message
Installation

[ ] I installed the plugin via zip from the Releases page
[X] I´m using the Netflix Repo
[ ] I´m using a different source (Please tell which)

Operating System

[X] Linux (x86/x64)
[ ] OSX (x86/x64)
[ ] Windows (x86/x64)
[ ] Linux (ARM)
[ ] Android

Additional informatin on the environment
using Libreelec with nightly milhouse build
Debug log
https://paste.ubuntu.com/p/YfrPGGRPN3/
Thanks for any help

repo server seems to be broken.
You can download addon from here: https://github.com/asciidisco/plugin.video.netflix/releases
and install in kodi with "Install from zip file"

authors: Shaboobala liberty-developer


text:
block in int_to_roman': undefined method divmod' for nil:NilClass (NoMethodError)

This occurs when a document includes a file that has tags without specifying tag filtering include::foo[]
Unfortunately, I haven't been able to distill to a simple test case.
/Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor/helpers.rb:225:in `block in int_to_roman': undefined method `divmod' for nil:NilClass (NoMethodError)
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor/helpers.rb:224:in `each'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor/helpers.rb:224:in `map'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor/helpers.rb:224:in `int_to_roman'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor/section.rb:118:in `sectnum'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor/section.rb:116:in `sectnum'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/ext/asciidoctor/section.rb:21:in `numbered_title'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3042:in `block (2 levels) in layout_toc_level'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3791:in `block in theme_font'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/prawn-2.2.2/lib/prawn/font.rb:62:in `block in font'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/prawn-2.2.2/lib/prawn/font.rb:215:in `save_font'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/prawn-2.2.2/lib/prawn/font.rb:60:in `font'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/ext/prawn/extensions.rb:208:in `font'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3790:in `theme_font'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3041:in `block in layout_toc_level'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3039:in `each'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3039:in `layout_toc_level'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3101:in `block (2 levels) in layout_toc_level'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/prawn-2.2.2/lib/prawn/document/bounding_box.rb:286:in `indent'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/prawn-2.2.2/lib/prawn/document.rb:505:in `indent'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3100:in `block in layout_toc_level'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3039:in `each'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3039:in `layout_toc_level'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:3025:in `layout_toc'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:2977:in `block in allocate_toc'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/ext/prawn/extensions.rb:874:in `instance_exec'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/ext/prawn/extensions.rb:874:in `block in dry_run'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/prawn-2.2.2/lib/prawn/font.rb:62:in `block in font'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/prawn-2.2.2/lib/prawn/font.rb:215:in `save_font'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/prawn-2.2.2/lib/prawn/font.rb:60:in `font'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/ext/prawn/extensions.rb:208:in `font'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/ext/prawn/extensions.rb:873:in `dry_run'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:2976:in `allocate_toc'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:218:in `block in convert_document'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:596:in `indent_section'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:214:in `convert_document'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/lib/asciidoctor/pdf/converter.rb:138:in `convert'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor/document.rb:1141:in `convert'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor.rb:1515:in `convert'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor.rb:1589:in `block in convert_file'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor.rb:1589:in `open'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor.rb:1589:in `convert_file'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor/cli/invoker.rb:129:in `block in invoke!'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor/cli/invoker.rb:112:in `each'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-1.5.7.1/lib/asciidoctor/cli/invoker.rb:112:in `invoke!'
	from /Users/cer/.rvm/gems/ruby-2.4.1/gems/asciidoctor-pdf-1.5.3/bin/asciidoctor-pdf:27:in `<top (required)>'
	from /Users/cer/.rvm/gems/ruby-2.4.1/bin/asciidoctor-pdf:23:in `load'
	from /Users/cer/.rvm/gems/ruby-2.4.1/bin/asciidoctor-pdf:23:in `<main>'
	from /Users/cer/.rvm/gems/ruby-2.4.1/bin/ruby_executable_hooks:15:in `eval'
	from /Users/cer/.rvm/gems/ruby-2.4.1/bin/ruby_executable_hooks:15:in `<main>'

I can't figure out the code path. Do you happen to be using any extensions that might be adding a section to the document programmatically?

No.

Hm. Strange. Let me know when you can reduce this down to an example you can share (or point me to source you're converting).

I think this is (or was) a bug in the Asciidoctor core 1.5.x release line. Since Asciidoctor 2, tghe part numeral is no longer generated during the call to Section#sectnum, but rather eagerly when the part is added to the document tree. I think when that change was made, this issue was fixed. Please reopen if you can reproduce using Asciidoctor 2.

authors: cer mojavelinux

repo: asciidoctor/asciidoctor-pdf


text:
test(use-case/nodecli): スナップショットテストを追加

nodecliのsrcに対するスナップショットテストを追加した

pretestでまとめて npm install
execしてそのstdoutをスナップショットとして保存して比較する

ライブラリアップデート的にもこういうのないと確認が面倒そうと思ったので、追加

Deploy preview for js-primer ready!
Built with commit c7275c629522ef39853f51df9ec01117e87574b9
https://deploy-preview-1123--js-primer.netlify.com

#1122 を試してて commander@5は実装してないフラグを渡すとエラーとなるように改善されてた

とりあえず先にマージします

source: gharchive/pull-request

authors: azu bot-user

repo: asciidwango/js-primer


text:
implement NoTLS (--plaintext) support

This PR implements:

topaz service support for running the gRPC services without TLS
topaz CLI support for --plaintext

For the service part, when:
certs block entries contain empty string(s), like:
certs:
  tls_key_path: ''
  tls_cert_path: ''
  tls_ca_cert_path: ''

or
the certs block contains no entries like:
certs:

or
when no certs block is present
the gRPC service instance will start without the associated TLS credentials.
The topaz CLI add the --plaintext or -P parameter to indicate that the gRPC connection will use plain-text HTTP/2 when connecting to server (no TLS).
  -P, --plaintext                use plain-text HTTP/2 (no TLS) ($TOPAZ_PLAINTEXT)

When using grpcurl or grpcui use the -plaintext command line flag.
This PR also changes the default security mode of the health and metrics services to by default run with NoTLS.

Pull Request Test Coverage Report for Build 11827984865
Details

35 of 734 (4.77%)  changed or added relevant lines in 24 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.3%) to 8.188%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

cmd/topaz/main.go
0
1
0.0%

pkg/app/authorizer.go
0
1
0.0%

pkg/cli/cmd/directory/backup.go
0
1
0.0%

pkg/cli/cmd/directory/exporter.go
0
1
0.0%

pkg/cli/cmd/directory/manifest.go
2
3
66.67%

pkg/cli/cmd/directory/restore.go
0
1
0.0%

pkg/cli/cc/health.go
0
2
0.0%

pkg/cli/clients/authorizer/client.go
11
13
84.62%

pkg/cli/clients/directory/client.go
13
15
86.67%

internal/pkg/service/builder/service.go
0
3
0.0%

Files with Coverage Reduction
New Missed Lines
%

pkg/cc/config/config.go
1
0.0%

Totals

Change from base Build 11788226662:
-0.3%

Covered Lines:
791

Relevant Lines:
9661

💛  - Coveralls

authors: coveralls gertd

repo: aserto-dev/topaz


text:
Remove channels parameter which limits payment channels shown

Passing the parameter channels: ["card", "bank"] is limiting the payment channels on checkout and not showing the USSD, and Visa QR options.
Removing this parameter allows merchants to have all channels by default. They can still pass in channels of choice if they need to restrict.

This has been fixed and published

authors: ashinzekene steveamaza


text:
4.2 breaks on build: #if targetEnvironment(simulator)

Installed via cocoapods v1.5.3
Xcode 9.2
deployment target: 11.0
v 4.1.0 works fine; 4.2.0 won't build (see attached)
Reachability.swift line 105 and 248:

https://github.com/ashleymills/Reachability.swift/issues/285 was meant to resolve a warning in xcode 9.3, but the implemented change breaks things in 9.2; a warning in 9.3 is far better than broken in 9.2...

You can use an old Reachability release if you're using an old version of Xcode.

True, and for now I will. At the very least this needs to be documented though.

@ashleymills works for me — appreciate your efforts developing and maintaining this!

authors: bbaic hipwelljo


text:
Use script to generate market data for binance, and exchange function for lot sizing

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Feature

What is the current behavior? (You can also link to an open issue here)

Product entries are done by hand, for exchanges with many assets this is tedious and prone to human error.
getMinimumSize on portfolioManager doesn't handle Binance's lot size requirements correctly result in occasional errors when posting small orders.

What is the new behavior (if this is a feature change)?

Binance market data is now moved to a second .js file, which is updateable with a script. The script parses the product definitions for all available pairs and adds them to the product list with all necessary information.
Added the function getLotSize to the exchange and updated portfolioManager to check for this function and use it is available for computing the correct lot size of an order. General rules are that if we are blow the minimum, we don't trade. This will allow any exchange to define necessary behaviour for lot sizing to that specific exchange.

Other information:

If we are happy with this approach for managing market data I'll apply the change to Kraken and Bitfinex as well.

This change is built on top of #1508

I submitted the same change for Kraken, as a bonus I was finally able to remove all the special handling code for newer assets. While not clearly documented in their AI, the returned API object keys actually give us the correct order book name for an asset pair queries.

@cmroche
Tested although got the following error:

2017-12-26 16:07:17 (DEBUG):	[binance.js] (addOrder) BUY 1704.8631578947368 POE @0.00000189 BTC
2017-12-26 16:07:18 (ERROR):	[binance.js] (addOrder) returned an irrecoverable error: undefined
2017-12-26 16:07:18 (DEBUG):	[binance.js] entering "setOrder" callback after api call, err: [binance.js] undefined data: {"code":-1013,"msg":"Filter failure: LOT_SIZE"}

@ms121 The undefined error suggst that you might be using a binance older than 1.1.0, could you check and confirm that please?
That aside, I'll look into why it tried to allow that order to go through, it clearly should have been prevented due to the lot size.

@ms121 Should be fixed now.

@cmroche Sorry for another newb question, but what is the easiest way to download all the changed files in the "files changed" section, or otherwise update my gekko installation with these latest code changes? I've been copying and pasting the text, but I'm thinking there has to be a better way. Thanks!

@jessrbird You cannot really download just the changed files, the easiest way to test a PR is to checkout or download the branch it is based on. In this case that would be here: https://github.com/cmroche/gekko/tree/origin/product_updates

I pulled this commit and ran the updateBinance.js script, the markets file looks correct.  When running it on actual trading I get the error:

2017-12-28 10:10:54 (DEBUG):    [binance.js] entering "setBalance" callback after api call, err: null, data: [object Object]
2017-12-28 10:10:54 (ERROR):    Wanted to buy ZEC but the amount is too small (0.000000000000) at binance

@werkkrew Please update your binance module with npm i binance@1.1.1 to fix the error reporting. I'll look at why it tried to buy with nothing.

@werkkrew The error that you are seeing, that the amount was too small, is correct if the amount it is requesting is too small for the market LOT size. I've made some small changes though, changing the error to a warning and printing the original requested amount and price (not the adjusted) just in case the lot size calculation was wrong. If you update try again and let me know if the requested amount was actually larger than the minimum lot sizes.

@cmroche I downloaded the zip using the link you provided above to get the latest PR. Getting an error running tradebot at binance USDT/BCC pair. The bot tried to sell BCC, but BCC balance was already 0. Here is the error:
/home/money/gekko/plugins/trader/portfolioManager.js:236
return log.warning(
^
TypeError: log.warning is not a function
at process (/home/money/gekko/plugins/trader/portfolioManager.js:236:18)
at bound (/home/money/gekko/node_modules/lodash/dist/lodash.js:729:21)
at Trader.getLotSize (/home/money/gekko/exchanges/binance.js:215:12)
at Trader.bound [as getLotSize] (/home/money/gekko/node_modules/lodash/dist/lodash.js:729:21)
at Manager.sell (/home/money/gekko/plugins/trader/portfolioManager.js:261:19)
at Manager.bound [as sell] (/home/money/gekko/node_modules/lodash/dist/lodash.js:729:21)
at Manager.act (/home/money/gekko/plugins/trader/portfolioManager.js:169:16)
at bound (/home/money/gekko/node_modules/lodash/dist/lodash.js:729:21)
at /home/money/gekko/node_modules/async/dist/async.js:3853:9
at /home/money/gekko/node_modules/async/dist/async.js:484:16
RECEIVED ERROR IN GEKKO 352640273213704
Let me know what you think.

@jessrbird Bah, should be log.warn sorry.. will send a fix in a few minutes

@cmroche Going to try starting the trading bot now, but I noticed a problem. I can no longer import data from binance. Other exchanges seem to work. I can't find any errors or logs, but when I go to scan available data, I see:
Unable to find datasets in the following markets:

binance:USDT:BCC
binance:USDT:BTC

@jessrbird Binance changed their about about 72 hours ago, this broke importing. PR #1568 is a fix for that problem.

authors: cmroche jessrbird ms121 werkkrew

repo: askmike/gekko


text:
Help a user who forgot to update their pnpm-lock.yaml

We could have some repository rule to re-run pnpm install --lockfile-only and verify that the users file is up-to-date.
Otherwise you might add a first-party dep to your package.json and then Bazel doesn't see that change.

@alexeagle Is this addressed by #157?

Resolved with update_pnpm_lock feature in npm_translate_lock

authors: alexeagle cgrindel gregmagolan


text:
Prevent null-ref and log exceptions form Serializer

Fixes #64.
@rynowak is this what you had in mind?

:watch: - yes this is the right idea. Needs a test

:up::date:

:shipit:

authors: ryanbrandenburg rynowak

repo: aspnet/Antiforgery


text:
Areas in .net core 3 which is using endpoints

In .net core 3, endpoints are used to for routing. Could you update the docs.

Can you create an issue from the doc page? That will connect the doc to the issue. Go to the bottom of the page and select the Content feedback issue.

authors: Rick-Anderson koppular


text:
This does not seem to work for my project.

I Added
services.AddControllersWithViews().AddRazorRuntimeCompilation();
and added the nuget package.
I am using Core 3.0 in an asp.net mvc application while trialling a Blazor component.
If I change a class on an element in a cshtml file, save and refresh the page in the browser nothing changes.  If I pause the application, make the change, then run the application I see the message applying changes in the lower left... then I get the error box stating that I need to edit or stop the application due to the changes.  A little slow to see minor changes to the view :)  Any help would be great.  Thank you

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8a83deb2-e035-bb66-be4f-68709bcad341
Version Independent ID: 16438f35-f35d-c097-0903-ec17e83e051e
Content: Razor file compilation in ASP.NET Core
Content Source: aspnetcore/mvc/views/view-compilation.md
Product: aspnet-core
Technology: aspnetcore-mvc
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Dup of #14800

authors: Rick-Anderson mikedcooper


text:
Whole-topic versioning

Fixes  #16907
Versioning for the remaining fifteen that I'm ms.author on. No warnings/errors on the build report.
Break out the EYE DROPS! 👁

@wadepickett  can you review this builds? Get two browsers side by side. One browser the review code, the other on the live site. Compare the versions too.

Thanks! Breaking out the eye drops...  :)

I did a visual check plus a text diff for all 16 x every version.  All looks good. Congrats! I could not find an unintended difference.   The only differences I did see were intended and checked out, such as:

A bug fix in fundamental/servers for
options.AllowSynchronousIO = true;
A repeated line edited out in iis/index
Sample download moved below the intro so the intro is first as it should be for caching/middlewhere.
The providers section of performance/response-compression changed so the intro rather than the code is first as it should be.

authors: Rick-Anderson guardrex wadepickett


text:
InsufficientExecutionStackException during model binding

Describe the bug
During a GET request to an MVC action, an exception of type InsufficientExecutionStackException was thrown.
Insufficient stack to continue executing the program safely. This can happen from having too many functions on the call stack or function on the stack using too much stack space.

System.InsufficientExecutionStackException: Insufficient stack to continue executing the program safely. This can happen from having too many functions on the call stack or function on the stack using too much stack space.
   at void System.Runtime.CompilerServices.RuntimeHelpers.EnsureSufficientExecutionStack()
   at bool Microsoft.AspNetCore.Mvc.ModelBinding.Validation.ValidationVisitor.Visit(ModelMetadata metadata, string key, object model)
   at bool Microsoft.AspNetCore.Mvc.ModelBinding.Validation.ValidationVisitor.Validate(ModelMetadata metadata, string key, object model, bool alwaysValidateAtTopLevel)
   at void Microsoft.AspNetCore.Mvc.ModelBinding.ObjectModelValidator.Validate(ActionContext actionContext, ValidationStateDictionary validationState, string prefix, object model, ModelMetadata metadata)
   at void Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.EnforceBindRequiredAndValidate(ObjectModelValidator baseObjectValidator, ActionContext actionContext, ParameterDescriptor parameter, ModelMetadata metadata, ModelBindingContext modelBindingContext, ModelBindingResult modelBindingResult)
   at async Task<ModelBindingResult> Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, object value)
   at void Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider+<>c__DisplayClass0_0+<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
   at async Task Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at async Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync()

The action is defined as thus:
[HttpGet]
[Route("{id}", Name = "MyRoute")]
[Produces("application/json", Type = typeof(MyViewModel))]
[ProducesResponseType(typeof(MyViewModel), StatusCodes.Status200OK)]
[ServiceFilter(typeof(IdValidationFilterAttribute))]
public async Task<ActionResult<MyViewModel>> Index(string id, CancellationToken cancellationToken = default)
{
    //
}

This issue looks similar to aspnet/Mvc#7357.
To Reproduce
Unknown, occurred during a normal GET request to a JSON resource in an ASP.NET MVC 2.2.0 application.
Expected behavior
Request succeeds.
Additional context
Application uses ASP.NET Core 2.2.0 and in-process IIS hosting.

@martincostello what's CompatibilityVersion in your application set to? In addition, are these the only two parameters being bound? For instance, it's not clear what IdValidationFilterAttribute does. A repro app would be super useful

We have CompatibilityVersion.Version_2_2 set.
The code for the filter is as below:
namespace MyApp
{
    public sealed class IdValidationFilterAttribute : ActionFilterAttribute
    {
        public override void OnActionExecuting(ActionExecutingContext context)
        {
            if (context.ActionArguments.TryGetValue("id", out object value) &&
                value is string id &&
                long.TryParse(id, NumberStyles.Integer, CultureInfo.InvariantCulture, out long _))
            {
                context.Result = new NotFoundResult();
                context.HttpContext.Response.StatusCode = 404;
                return;
            }

            base.OnActionExecuting(context);
        }
    }
}

We've only had one instance of this exception in the last 30 days, which was today, so I'm not sure what we could do to repro it with a simple app at the moment.

That's super weird. The error from https://github.com/aspnet/Mvc/issues/7357 is fairly consistent for a given model type and neither of the two parameter types from your action fit the bill. I'll go over the code once over to see we don't have any latent caching bugs in our code that might produce this exception.

I've just realised from further review that this error isn't coming from the action associated with the URL, but from the error handler for that URL. I noticed that from the ActionName in the logs:
MyApp.Controllers.ErrorController.Error (MyApp)

The action for the error handler is as thus:
[IgnoreAntiforgeryToken]
[Route("/error", Name = "Error")]
public IActionResult Error([FromQuery] int? id = null)
{
}

and error handling is configured like this:
app.UseExceptionHandler("/error")
    .UseStatusCodePagesWithReExecute("/error", "?id={0}");

Could it be that if the original action throws an exception, the model binding for the re-execute for the error page is getting stuck in a loop of some sort?

So the recommended solution is to downgrade to 2.1.6?

@chaim1221 No, you can use out-of-process hosting with 2.2.x instead.

So I'm not sure what you mean by out of process hosting. We are already using NginX as a proxy to Kestrel and we're still getting the error.
looking at
https://github.com/aspnet/Mvc/commit/522006d2c866add522a546aa3b4720577f1cee57#diff-4ac01a4b903e16c81e0386bf764ecc09R40
It appears I have to have the compatibilityVersion set to 2_2. We have it set to 2_1.
Thanks for the heads up re: vulnerabilities, although I definitely have no desire to go back to 2.1!
:)

The different IIS hosting models are explained here. This issue is specific to using the IIS in-process hosting with 2.2.x.
If you’re using nginx (presumably on Linux) then whatever issue you’re experiencing is something else.

I see. Well it definitely fits aspnet/Mvc#7357 --and yeah, we aren't using IIS, we are 100% Debian (also in a Docker container running NginX and Kestrel). It's working nicely, btw. :)

authors: chaim1221 martincostello pranavkm

repo: aspnet/AspNetCore


text:
Update MVC dependencies

Update MVC dependencies to react to routing name change in https://github.com/aspnet/AspNetCore/pull/4416

We're adding required PR checks to ensure the repo compiles before you merge. Can you update this PR by rebasing on master?
git fetch origin master:master
git rebase master

Alternative: if you plan to 'squash merge' this PR, you can also use a merge to update this PR. This can be easier to manage if you have lots of changes in this PR.
git fetch origin master:master
git rebase master

Done

authors: JamesNK natemcmaster


text:
Add package conflict manifest data to the targeting pack

Example file this generates:
Microsoft.AspNetCore.App.PackageOverrides.txt
Part of https://github.com/aspnet/AspNetCore/issues/6501
cc @nguerrera @dsplaisted @dagood

File looks great. Filed https://github.com/dotnet/sdk/issues/2937 to consume it.

Per @pakrym's suggestion, switched this to use ReferencePath instead. This should better align with what RAR does.
Result:
Microsoft.AspNetCore.App.PackageOverrides.txt

Doesn't this version assume all package references are 1:1 with assemblies? Is that guaranteed by anything?

It only makes that assumption for assemblies which build in this repo. And for this subset of stuff, yes, package id == assembly name.
Worth nothing - this list includes package IDs for assemblies we shipped as OOB packages in 2.x, but have dropped in 3.0. For instance, Microsoft.AspNetCore.Mvc.Razor|3.0.0-preview3-t000. We haven't, and never plan to, ship Microsoft.AspNetCore.Mvc.Razor for 3.0.

I'm seeing stuff that is not in the repo in the generated file. Isn't it the project references that are in this repo?

Newtonsoft.Json|12.0.1
System.IO.Pipelines|4.6.0-preview.19106.2
System.Runtime.CompilerServices.Unsafe|4.6.0-preview.19106.2
System.Security.AccessControl|4.6.0-preview.19106.2
System.Security.Cryptography.Cng|4.6.0-preview.19106.2
System.Security.Cryptography.Xml|4.6.0-preview.19106.2
System.Security.Permissions|4.6.0-preview.19106.2
System.Security.Principal.Windows|4.6.0-preview.19106.2
System.Text.Encodings.Web|4.6.0-preview.19106.2

Unless I'm misunderstanding, this would generate dupes if a package had more than one assembly:
 <_AspNetCoreAppPackageOverrides Include="@(ReferencePath->'%(NuGetPackageId)|%(NuGetPackageVersion)')" Condition=" '%(ReferencePath.NuGetPackageId)' != 'Microsoft.NETCore.App' AND '%(ReferencePath.NuGetSourceType)' == 'Package' " />

Not sure there are any such packages today or if dupes will break anything, but it seems like something I'd question if I ever had to revisit this code.

Worth noting - this list includes package IDs for assemblies we shipped as OOB packages in 2.x, but have dropped in 3.0. For instance, Microsoft.AspNetCore.Mvc.Razor|3.0.0-preview3-t000. We haven't, and never plan to, ship Microsoft.AspNetCore.Mvc.Razor for 3.0.

I think this is fine. This says that anything less than that version is overridden by this package, which is true.

@natemcmaster We're thinking that data files for targeting and runtime packs should go in the data/ folder of the package.  So I'd suggest that the path for this file should be data/PackageOverrides.txt.

I'm seeing stuff that is not in the repo in the generated file.

That's intentional. We include some assemblies like System.IO.Pipelines in the aspnet targeting pack. They are built by corefx, but for reasons, they didn't want it in microsoft.netcore.app.

My point is that it looks like you are assuming things outside this repo have 1 package:1 assembly. I doubt that will change in practice. But that was my point.

Makes sense. So far, this assumption has been correct for all assemblies currently in the targeting pack. We don't really have a good way to test this, so it could break at some point 😬

Per your request, I've updated the PR to put the manifest in data/PackageOverrides.txt

authors: dsplaisted natemcmaster nguerrera


text:
Updated all ASP.NET 4.x Identity topics to include ASP.NET 4.x in tit…

…le metadata

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: wadepickett sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Updated the ASP.NET 4.x Identity topics to include ASP.NET 4.x in their title.  Per branding discussion with Product Unit and this helps clarify what version these are for which was causing some SEO issues.  Part of april seo effort.

authors: dnfclas wadepickett

repo: aspnet/AspNetDocs


text:
'Router' cannot find any component with a route for '/index.html'

Hi, first of all - thank you for your brilliant efforts in blazor!
Now question.
I try to run sample app on our server. But it doesn't have index.html in the root and complete blazor bundle is located in the subfolder:

and I come to the point

Changing internal links like this didn't help:

So, what can be the option?

I have left all @page... syntax like it was in original bundle.
Used .NET Core "Publish":

The only thing I have changed inside the output of index.html is

my publish url is going to be:
https://mysrv:27750/modules.getfz.GM_PersonalDevice/web.zip/blazor3/index.html
and still receive an error:

Is there anything else to setup?

Your URL cannot end with /index.html, at least not given the built-in conventions of the router. It needs to end either with a plain / (which maps to your Index component) or with the name of a component (e.g., /mycomponent).

Thank you, very cool. I could handle loading of the first index page (I made it look like the "fetching" page) by manual feeding  index.html  as an HTTP response, when URL ends with blazor../

Next step:
It is not clear which content should server return when e.g link "/counter" is pressed? There is no other valid content except index.html.
In sample web-server, after clicking "/counter" nav-link, url is changed and in HTTP-Requests from the browser I see request of favicon.ico only.

But on my server after pressing "/counter" I see real request of the "/counter" and it is not clear which content should be sent back to the browser as an HTTP response?

I think I got, completely works:
all .cshtml pages should have the same "@page" links declarations like in example:

but in navigation menus all should be changed:

Result:

Thank you very much!!!

authors: SteveSandersonMS hhblaze

repo: aspnet/Blazor


text:
React to EF rename

:watch: https://github.com/aspnet/EntityFramework/pull/3331

:shipit:

authors: Eilon natemcmaster

repo: aspnet/Diagnostics


text:
When i run, error

It said:
Startup.cs(35,50): error CS1061: 'DbContextOptionsBuilder' does not contain a definition for 'UseInMemoryDatabase' and no accessible extension method 'UseInMemoryDatabase' accepting a first argument of type 'DbContextOptionsBuilder' could be found (are you missing a using directive or an assemblyreference?) [C:\Users\Peter\Desktop\aspnetcoreapp\aspnetcoreapp.csproj]
The build failed. Please fix the build errors and run again.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 583ae3e7-5915-f22b-e54c-6c100cf2a7da
Version Independent ID: 95f2549e-29f9-e31a-b4d0-432b971643ff
Content: Tutorial: Create a web API with ASP.NET Core MVC
Content Source: aspnetcore/tutorials/first-web-api.md
Product: aspnet-core
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Great error message. You might want to compare with the download.

repo: aspnet/Docs


text:
Authorization Filters

ASP.NET MVC -> Security -> Authorization Filters

Any status update on when this will be written?

Not yet - it's still on the backlog.

Hi Dan. Thanks for the update.

I hope this stays:
[ClaimsPrincipalPermission(SecurityAction.Demand, Operation = "OperationA", Resource = "DeletePage")]
or improves.
At the moment I am unable to find anything that parallels it. I have been searching all of 30 minutes though.

@blowdart

Long gone. Use imperative resource based checks. It was never part of asp.bet in the first place, it was code access security based which has been removed.

I hope this stays:

What @blowdart is referring to is something similar added to ASP.NET 5, but much improved. Check out the updated docs for more info (policy-based and resource-based authrization).

https://docs.asp.net/en/latest/security/authorization/resourcebased.html

Merging with #119

This issue is linked to here https://docs.microsoft.com/en-us/aspnet/core/security/authorization/authorization-filters
It seems like it's been closed incorrectly as  #119 does have a little bit about authorization filters but it doesn't cover it properly and the dedicated page has been left as a stub.
It seems either this issue should be re-opened or the stub should be removed.

@rtpHarry I think we just need to remove the stub. Most of the authorization related content is actually covered under the Authorization section, which is linked to from the Filters doc: https://docs.microsoft.com/en-us/aspnet/core/security/authorization/introduction.
Are there specific concerns related to authorization filters that you would like to see added to the docs?

I don't know about them, I'm just learning. I just wanted to point it out but I don't have a solution :)

@rtpHarry That's cool. Let us know if you have any specific questions!

authors: blowdart brockallen danroth27 neville-lewis rtpHarry uwbadgerdev


text:
TIL: libuv1 is now on wily

Not sure if this has some positive implications for the linux/ubuntu install instructions.  Instead of having to compile libuv, people should be directed to apt-get it.
https://github.com/libuv/libuv/issues/643#issuecomment-164766351

Our install instructions were written with 14.04 in mind. Look here
This is definitely worth re-visiting there is a new LTS version of Ubuntu which I believe is April of 2016.
Feel free to re-open then or if you have a compelling reason that we should do it now.

Compelling alternative would be to support both.

authors: atrauzzi shirhatti


text:
Add XSS documentation

Addresses #90
@Rick-Anderson, @Tratcher, @NTaylorMullen please review.

:shipit:

@blowdart can you make the small edits from @tratcher ?

@blowdart  consider editing  index.rst and move this before auth topics - it's very important and buried  at the bottom.

Disagree with index ordering - this is mostly automatic, whereas authn & z take more thought.

Are we all happy?

@blowdart  I need 15 more minutes

:shipit:

@spboyer Everyone is happy. Everything is awesome.

@rick-Anderson is happy :-)

:watch:

authors: NTaylorMullen Rick-Anderson Tratcher blowdart spboyer


text:
Consolidated two different links to the same article, into 1 link

There were two links to the same article.... with slightly different title... i merged the two links into one, without losing any information (i hope). cheers.

Hi @secretGeek, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
We will now validate the agreement and then real humans will evaluate your PR.
TTYL, DNFBOT;

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

aspnetcore/publishing/index.md
:white_check_mark:Succeeded

Details

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@secretGeek  nice fix and excellent blog post.

Cheers 😀

authors: Rick-Anderson bradygaster dnfclas secretGeek


text:
Update raw-sql.md

tiny typo

Thanks!

authors: ajcvickers mogaozq


text:
Reformat

Fix indention, apostrophes and newlines.
Also fixes some typos like a / after IDictionary or too many curly brackets.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: Danghor sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

@Danghor  we can't review this until you sign the CLA.

Thank you @Danghor!

authors: Danghor Rick-Anderson divega dnfclas


text:
CLI Commands: honor default namespace from project

Apparently, this is a thing in project.json.
{
  "tooling": {
    "defaultNamespace": "Not.Project.Name"
  }
}

We should honor that in our commands instead of just using the project name.

Not sure where this "tooling" node came from, but it is not in the project model for CLI projects.

cc @rowanmiller since I know we had resolved a few bugs as dupes of this one. We may want to have something (maybe repurpose this issue?) to make sure we handle the equivalent scenario for tooling RTM.

Agreed, reopening and removing milestone so that we can discuss in triage

Edited title to make it non project.json specific.

This will go away as we consolidate with MSBuild

Where is the "tooling" element supposed to go in project.json? I can't seem to find any documentation on this.
Also, what version of the CLI was it first supported in?

@NightOwl888  The  .NET Core SDK 1.0.0 no longer supports project.json or Visual Studio 2015. Everyone doing .NET Core development is encouraged to migrate from project.json to csproj and Visual Studio 2017.
BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.

I've been told (but have not confirmed) that VS 2017 and the unified .csproj file for .NET Framework/.NET Core is not fully functional yet. And I really don't have the time to migrate my 35 projects (105 different build configurations) to .csproj just to jump over a lot of other hurdles trying to develop again. So for the time being, we are sticking with the project.json file (since that is the way your Microsoft contributor set it up a few months ago).
That said, my question still applies and is not yet answered.

authors: NightOwl888 ajcvickers bricelam divega natemcmaster rowanmiller

repo: aspnet/EntityFramework


text:
Scaffold-DbContext throwing System.IO.FileNotFoundException error.

Updated my Visual Studio with RC 2 and created a Web API project.
Followed documentation guidelines for new application with existing database.
The following error was thrown hitting Scaffold-DbContext.
System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=7.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The system cannot find the file specified. File name: 'Newtonsoft.Json, Version=7.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'
    at Microsoft.DotNet.ProjectModel.ProjectReader.ReadProject(Stream stream, String projectName, String projectPath, ProjectReaderSettings settings)
   at Microsoft.DotNet.ProjectModel.ProjectReader.GetProject(String projectPath, ProjectReaderSettings settings)
    at Microsoft.EntityFrameworkCore.Tools.DispatchCommand.<>c__DisplayClass2_0.<Create>b__0()
    at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Microsoft.EntityFrameworkCore.Tools.Program.Main(String[] args)
 Could not load file or assembly 'Newtonsoft.Json, Version=7.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The system cannot find the file specified.

Got a similiar Newtonsoft related error using CLI dotnet ef.

Can you include your project.json please?

I posted a question on Stack Overflow. It's not the complete project.json file, but all the relevant parts should be there. If you would like to see the complete file posted here I can do that too.

Can you try adding "Newtonsoft.Json": "7.0.1" to your dependencies and let me know if that works?

I tried adding Newtonsoft.Json: 7.0.1. I got errors with other dependencies were looking for version 8.03. I got messages about some things looking for 8.03, "and got 7.01 instead."

What happens if you add both?

When adding 7.0.1, or both 8.0.3 and 7.0.1 I get the following warnings:
Dependency conflict. Microsoft.AspNetCore.JsonPatch 1.0.0-rc2-final expected Newtonsoft.Json >= 8.0.3 but got 7.0.1
Dependency conflict. Microsoft.AspNetCore.Mvc.ViewFeatures 1.0.0-rc2-final expected Newtonsoft.Json >= 8.0.3 but got 7.0.1
Dependency conflict. Microsoft.Extensions.Configuration.Json 1.0.0-rc2-final expected Newtonsoft.Json >= 8.0.3 but got 7.0.1
The project builds with warnings, but errors when trying to use dotnet ef or Scaffold-DbContext.

Sorry - I forgot. You have to include "dncore50" inside the imports section for Microsoft.EntityFrameworkCore.Tools inside the tools section. This is a workaround for a problem with CLI in RC2 and should go away in RTM.

Bummer. Thought I had a remedy.
Still getting the errors with the following:
  "tools": {
    "Microsoft.EntityFrameworkCore.Tools": {
      "version": "1.0.0-preview1-final",
      "imports": [
        "portable-net45+win8+dnxcore50",
        "portable-net45+win8",
        "dnxcore50"
      ]
    }

I have tried variations of the above, but have not solved this riddle.

@mhenderson442 just in case it helps, what NuGet feed are you using? Do you have a project.lock.json file?

I am using nuget.org(https://api.nuget.org/v3/index.json).
I do have a project.lock.json.
There are two references to Newtonsoft.Json 7.0.1. Excerpted below.
      "Microsoft.DotNet.ProjectModel/1.0.0-rc2-002702": {
        "type": "package",
        "dependencies": {
          "Microsoft.CSharp": "4.0.1-rc2-24027",
          "Microsoft.Extensions.DependencyModel": "1.0.0-rc2-002702",
          "Newtonsoft.Json": "7.0.1",
          "NuGet.Packaging": "3.5.0-beta-final",
          "NuGet.RuntimeModel": "3.5.0-beta-final",
          "System.Dynamic.Runtime": "4.0.11-rc2-24027",
          "System.Reflection.Metadata": "1.3.0-rc2-24027",
          "System.Runtime.Loader": "4.0.0-rc2-24027",
          "System.Runtime.Serialization.Primitives": "4.1.1-rc2-24027",
          "System.Security.Cryptography.Algorithms": "4.1.0-rc2-24027",
          "System.Threading.Thread": "4.0.0-rc2-24027",
          "System.Xml.XDocument": "4.0.11-rc2-24027"
        },

      "Microsoft.Extensions.DependencyModel/1.0.0-rc2-final": {
        "type": "package",
        "dependencies": {
          "Microsoft.DotNet.InternalAbstractions": "1.0.0-rc2-002702",
          "Newtonsoft.Json": "7.0.1",
          "System.Diagnostics.Debug": "4.0.11-rc2-24027",
          "System.Dynamic.Runtime": "4.0.11-rc2-24027",
          "System.Linq": "4.1.0-rc2-24027"
        },

If it's useful, here are some the steps I follow trying to solve the issue:

Make a change to project.json and save
Go to cmd prompt and hit dotnet restore
If there are no errors or warnings, I hit dotnet ef

At this point I get the error you see at the start of this thread.

Unfortunately I think we are hitting #5320.

#5320 does look quite similar. I used the Web Api template to create the project. I even tried a Web Application template to see if something was not right with the template.

@natemcmaster do you think this in fact may be a dupe of #5320?

Depends. Is emitEntryPoint set to true?  Also, what is the output when adding --verbose?

@natemcmaster
emitEntryPoint is set to true.
  "buildOptions": {
    "emitEntryPoint": true,
    "preserveCompilationContext": true
  },

Here is the output when running Scaffold-DbContext with verbose:
Working directory: C:\Source\Repos\MercurySchool\MercurySchool\MercurySchoolApi
Executing command: dotnet ef --configuration Debug --build-base-path .\bin\ dbcontext scaffold 'Server=[Server named omitted];Data Source=[Data source omitted];Initial Catalog=MercurySchool;Persist Security Info=False;User ID=[User ID omitted];Password=[Password omitted];Pooling=False;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;' Microsoft.EntityFrameworkCore.SqlServer --context DbMercurySchool --output-dir DataModel --verbose
System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=7.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The system cannot find the file specified. File name: 'Newtonsoft.Json, Version=7.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'
    at Microsoft.DotNet.ProjectModel.ProjectReader.ReadProject(Stream stream, String projectName, String projectPath, ProjectReaderSettings settings)
   at Microsoft.DotNet.ProjectModel.ProjectReader.GetProject(String projectPath, ProjectReaderSettings settings)
   at Microsoft.EntityFrameworkCore.Tools.DispatchCommand.<>c__DisplayClass2_0.<Create>b__0()
   at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
    at Microsoft.EntityFrameworkCore.Tools.Program.Main(String[] args)
 Could not load file or assembly 'Newtonsoft.Json, Version=7.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The system cannot find the file specified.

@mhenderson442 thanks for that. I suspect this is related to imports. Try this to fix it

Delete %USERPROFILE%\.nuget\packages\.tools\Microsoft.EntityFrameworkCore.Tools and %USERPROFILE%\.nuget\packages\Microsoft.EntityFrameworkCore.Tools.
Make sure your "tools" section matches this:

  "tools": {
    "Microsoft.EntityFrameworkCore.Tools": {
      "version": "1.0.0-preview1-final",
      "imports": "portable-net451+win8"
    }

cref https://docs.efproject.net/en/latest/cli/dotnet.html
3. Re run dotnet restore and look in the output for errors. VS will trigger this automatically when you save project.json and spits output to "Output > Package Manager".
4. If commands still fail, can you share the files in %USERPROFILE%\.nuget\packages\.tools\Microsoft.EntityFrameworkCore.Tools?

@natemcmaster Following you instructions, dotnet ef worked.
Using Package Manager Console I got the following error:
Scaffold-DbContext : The term 'Scaffold-DbContext' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a
path was included, verify that the path is correct and try again.
At line:1 char:1
+ Scaffold-DbContext -Provider "Microsoft.EntityFrameworkCore.SqlServer ...
+ ~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Scaffold-DbContext:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

command otnet ef dbcontext scaffold executed with errors, but I think that's unrelated to this issue.
Got the following error: One for more errors ocurred. (Invalid object name 'sys.sequences'.).
I am connecting to a SQL Azure database. Is this supported?

:+1: glad we got the first error out of the way.

dotnet ef dbcontext scaffold executed with errors, but I think that's unrelated to this issue.

Yes, can you open another issue? We support querying from SQL Azure, but I don't think we have ever testing scaffolding from a SQL Azure database. cc @lajones
This can probably be closed now. For anyone else who encounters this (cc @Kurira), the "Newtonsoft.Json" error is likely due to NuGet caching issues. Clear your caches as instructed above  https://github.com/aspnet/EntityFramework/issues/5458#issuecomment-221912806

From perspective I think this issue is closed. I will tinker some more to make sure the other errors warrant a new issue.
@lajones @divega @natemcmaster Thanks for the guidance.

Hello,
I came to a similar problem after fixing Could not load file or assembly 'Newtonsoft.Json issue.
Now I get: Could not load file or assembly 'System, Version=2.0.5.0 [...]
My project.json
{
  "dependencies": {
    "Microsoft.AspNetCore.Diagnostics": "1.0.0-rc2-final",
    "Microsoft.AspNetCore.Mvc": "1.0.0-rc2-final",
    "Microsoft.EntityFrameworkCore.SqlServer": "1.0.0-rc2-final",
    "Microsoft.EntityFrameworkCore.Tools": {
      "version": "1.0.0-preview1-final",
      "type": "build"
    },
    "Microsoft.AspNetCore.Razor.Tools": {
      "version": "1.0.0-preview1-final",
      "type": "build"
    },
    "Microsoft.AspNetCore.Server.IISIntegration": "1.0.0-rc2-final",
    "Microsoft.AspNetCore.Server.Kestrel": "1.0.0-rc2-final",
    "Microsoft.AspNetCore.StaticFiles": "1.0.0-rc2-final",
    "Microsoft.Extensions.Configuration.EnvironmentVariables": "1.0.0-rc2-final",
    "Microsoft.Extensions.Configuration.Json": "1.0.0-rc2-final",
    "Microsoft.Extensions.Logging": "1.0.0-rc2-final",
    "Microsoft.Extensions.Logging.Console": "1.0.0-rc2-final",
    "Microsoft.Extensions.Logging.Debug": "1.0.0-rc2-final",
    "Microsoft.AspNetCore.Authentication.Cookies": "1.0.0-rc2-final",
    "Microsoft.AspNetCore.Authentication": "1.0.0-rc2-final",
    "Microsoft.AspNetCore.Identity": "1.0.0-rc2-final",
    "Microsoft.AspNetCore.Identity.EntityFrameworkCore": "1.0.0-rc2-final"
  },

  "tools": {
    "Microsoft.EntityFrameworkCore.Tools": {
      "version": "1.0.0-preview1-final",
      "imports": "portable-net45+win8"
    },
    "Microsoft.AspNetCore.Razor.Tools": {
      "version": "1.0.0-preview1-final",
      "imports": "portable-net45+win8+dnxcore50"
    },
    "Microsoft.AspNetCore.Server.IISIntegration.Tools": {
      "version": "1.0.0-preview1-final",
      "imports": "portable-net45+win8+dnxcore50"
    }
  },

  "frameworks": {
    "net461": {
      "frameworkAssemblies": {
      }
    }
  },

  "buildOptions": {
    "emitEntryPoint": true,
    "preserveCompilationContext": true
  },

  "publishOptions": {
    "include": [
      "wwwroot",
      "Views",
      "appsettings.json",
      "web.config"
    ]
  },

  "scripts": {
    "prepublish": [ "npm install", "bower install", "gulp clean", "gulp min" ],
    "postpublish": [ "dotnet publish-iis --publish-folder %publish:OutputPath% --framework %publish:FullTargetFramework%" ]
  }
}

Any ideas?

@ole1986 as the issue describes, your imports are incorrect. For preview1, use "portable-net451+win8" See https://github.com/aspnet/EntityFramework/issues/5458#issuecomment-221912806


text:
Composite keys with one auto incrementing field fails on AddRange()

Steps to reproduce
I'm trying to set up a table with a composite key, where one of the fields autoincrements. Here's the relevant code from the model and context classes:
In the model class:
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)] // Might not be needed, just trying to make it work
public int EventId { get; set; }

[Required]
public string UserId { get; set; }

In the DB context:
public DbSet<Event> Events { get; set; }
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<Event>()
        .HasKey(e => new { e.EventId, e.UserId });
}

The issue
In the application code, EventId is never directly set - I'm leaving that up to the database. When trying to insert multiple events (all of which have EventId=0, the default value) I get this error on the .AddRange() call:
Exception message: An exception of type 'System.InvalidOperationException' occurred in Microsoft.EntityFrameworkCore.dll but was not handled in user code

Additional information: The instance of entity type 'Event' cannot be tracked because another instance of this type with the same key is already being tracked. When adding new entities, for most key types a unique temporary key value will be created if no key is set (i.e. if the key property is assigned the default value for its type). If you are explicitly setting key values for new entities, ensure they do not collide with existing entities or temporary values generated for other new entities. When attaching existing entities, ensure that only one entity instance with a given key value is attached to the context.
Stack trace:    at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.IdentityMap`1.Add(TKey key, InternalEntityEntry entry)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.StartTracking(InternalEntityEntry entry)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntry.SetEntityState(EntityState oldState, EntityState newState, Boolean acceptChanges)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.EntityGraphAttacher.PaintAction(EntityEntryGraphNode node)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.EntityEntryGraphIterator.TraverseGraph(EntityEntryGraphNode node, Func`2 handleNode)
   at Microsoft.EntityFrameworkCore.DbContext.SetEntityStates(IEnumerable`1 entities, EntityState entityState)
   at <my application's code>

Further technical details
EF Core version: 1.0.1
Operating system: Windows 7
Visual Studio version:  VS 2015
Other details about my project setup: I've run database migrations (to postgres) with these classes, and the resulting SQL table looks correct - the primary key is a composite key of the two Ids, and the EventId field is set to autoincrement.
If I remove the composite key declaration in OnModelCreating(), then I can insert in bulk just fine, and the EventId is populated correctly.

Note for triage: I was able to reproduce this with the RTM release but it does not repro with the latest bits. I believe it is likely a duplicate of #5776.
@dligge You should be able to work around the issue by setting the EventId property to store generated after the HasKey call. For example:
modelBuilder.Entity<Event>(b =>
{
    b.HasKey(e => new { e.EventId, e.UserId });
    b.Property(e => e.EventId).ValueGeneratedOnAdd();
});

authors: ajcvickers dligge


text:
React to Caching package rename

Dependent on https://github.com/aspnet/Caching/pull/49/files

@muratg

:shipit:

authors: Praburaj davidfowl


text:
[TINY] Fix to #2599 - Unary expression visitation throws for certain 'Not (!)' subqueries

Fix is to add a null check for for cases where we can't translate the operand

@anpete

:shipit:

authors: anpete maumar


text:
Query: Add OrderBy for definite results

Resolves #5220
@roji - Let us know if there are more tests which are failing due to missing order by.

:shipit:

authors: anpete smitpatel


text:
Migration silently fails to set identity

Hi
I cant change a Primarey key from being not identity to be Identity. The migration file looks correct but column is still not Identity after db update. Looks like there is a workitem om codeplex aswell, se below.
https://entityframework.codeplex.com/workitem/509

Note to assignee: Having identity: true on the AlterColumnOperation is OK if the column was created with IDENTITY.

Triage: we are not planning to do this in EF6.x at this point but we will do it in EF Core 1.x.

authors: bricelam divega klamfeldt


text:
Evaluation of DbQuery.ToString() frequently times out in the debugger

DbQuery<T>.ToString() was overridden in order to provide a simple means to obtain the SQL translation for a LINQ query, both programmatically and at debug time. When it works property, it provides a very nice experience.
However DbQuery<T>.ToString() has been found to be one of the top sources of timeouts while the debugger performs function evaluation. In general, functions that take a long time to evaluate can lead to a very lousy user experience using VS:

During function evaluation, the debugger UI is blocked
Once the evaluation of a function times out, the debugger preventively blocks any further function evaluation

We believe the main reasons DbQuery<T>.ToString() can take a long time to evaluate (and possibly exceed function evaluation timeout period) are:

The first query executed over a particular model may require model and database initialization, which usually involves expensive computation and several database roundtrips
A complex query may take a long time to compile

The VS debugger team has recommended that we make changes in future versions of EF6.x. After discussing several options with them, there doesn't seem to be any way to preserve the optimal simplicity of the current experience for the best case scenario (when the method returns in a timely manner) and at the same time we improve the experience for the worst case scenario (when the method takes too long to evaluate).
In summary, the proposed changes are:
a. Add a read-only Sql property on DbQuery<T> which invokes ToString() and exposes the result
b. Add a call to Debugger.NotifyOfCrossThreadDependency() on model and database initialization
c. Possibly add Debugger.NotifyOfCrossThreadDependency() calls in expensive (but not reentrant or repetitively invoked) parts of query compilation, or alternatively use some heuristic to estimate the query complexity and invoke the method if the query falls above certain threshold.
d. Annotate DbQuery<T> (and its derived types) with an attribute that restores the default behavior for the debugger, e.g. something like [DebuggerDisplay(@"\{System.Data.Entity.Infrastructure.DbQuery\}")] (this is oversimplified, doesn't include the generic argument). Alternatively we could include a message there pointing to the new Sql property.
Brief explanation:
Function evaluation works by hijacking a thread from the application that is paused to execute code. Debugger.NotifyOfCrossThreadDependency() can be used in a property to indicate the debugger that the code is going down some slow path so it can abort the evaluation in a more orderly manner than with a timeout. E.g. on the locals window, it shows a message saying "The function evaluation requires all threads to run." and presents a "threads" glyph that users can click on to force the evaluation.
There is no way to get the glyph on the root node of an instance or on a method.
The resulting experience would look like this for a query x:

Made a single change to the proposal. Clearing up assignment and milestone so that we can reassign this in triage.


text:
GetHashCode(Expression obj) of ExpressionEqualityComparer does not support ExpressionType.Assign

Hi there,
I have a homemade mapping/projection library that I use to map/project between DB models and DTOs. In the expression tree that this library builds, there are a few assignments (i.e. Expression.Assign) being made in the logic to support mapping/projection of abstract classes to derived types (i.e. I have the abstract Field, and the concrete BooleanField entities and the abstract FieldDto, and the concret BooleanFieldDto DTOs).
Here is the relevant part in the resulting expression tree:
.If (
	$field .Is Application.Domain.Models.BooleanField
) {
	.Block(Application.Domain.Models.BooleanField $booleanfield) {
		$booleanfield = (Application.Domain.Models.BooleanField)$field;
		$x = (Web.Models.Schemas.FieldDto).New Web.Models.Schemas.BooleanFieldDto(){
			Properties = .New Web.Models.Schemas.BooleanFieldPropertiesDto(){
				DefaultValue = ($booleanfield.Properties).DefaultValue,
				Label = ($booleanfield.Properties).Label,
				Hint = ($booleanfield.Properties).Hint,
				Placeholder = ($booleanfield.Properties).Placeholder,
				IsRequired = ($booleanfield.Properties).IsRequired
			},
			Name = $booleanfield.Name,
			FieldTypeName = $booleanfield.FieldTypeName
		}
	}
} .Else {
	.Default(System.Void)
};

Mapping an existing entity model to its DTO counterpart works as expected, however doing a projection fails with the following exception:
An unhandled exception occurred while processing the request.

NotImplementedException: The method or operation is not implemented.

Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode(Expression obj)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode(Expression obj)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode<T>(IList<T> expressions)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode(Expression obj)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode<T>(IList<T> expressions)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode(Expression obj)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode(Expression obj)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode(Expression obj)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode(Expression obj)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode<T>(IList<T> expressions)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode(Expression obj)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode<T>(IList<T> expressions)
Microsoft.EntityFrameworkCore.Query.Internal.ExpressionEqualityComparer.GetHashCode(Expression obj)
Microsoft.EntityFrameworkCore.Query.CompiledQueryCacheKeyGenerator+CompiledQueryCacheKey.GetHashCode()
Microsoft.EntityFrameworkCore.Query.RelationalCompiledQueryCacheKeyGenerator+RelationalCompiledQueryCacheKey.GetHashCode()
System.Collections.Generic.ObjectEqualityComparer.GetHashCode(T obj)
System.Collections.Concurrent.ConcurrentDictionary.TryGetValue(TKey key, out TValue value)
Microsoft.Extensions.Caching.Memory.MemoryCache.TryGetValue(object key, out object result)
Microsoft.Extensions.Caching.Memory.CacheExtensions.TryGetValue<TItem>(IMemoryCache cache, object key, out TItem value)
Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore<TFunc>(object cacheKey, Func<Func<QueryContext, TFunc>> compiler)
Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddAsyncQuery<TResult>(object cacheKey, Func<Func<QueryContext, IAsyncEnumerable<TResult>>> compiler)
Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileAsyncQuery<TResult>(Expression query)
Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync<TResult>(Expression query, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync<TResult>(Expression expression, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync<TSource, TResult>(MethodInfo operatorMethodInfo, IQueryable<TSource> source, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.SingleOrDefaultAsync<TSource>(IQueryable<TSource> source, CancellationToken cancellationToken)
HedgeSoft.Web.Controllers.SchemasController+<Get>d__5.MoveNext() in SchemasController.cs
var schema = await _mapper.Project<SchemaDetailedDto>(

I noticed in the code of ExpressionEqualityComparer.GetHashCode(Expression obj), there is no case for ExpressionType.Assign, hence the NotImplementedException thrown above.
Is this something that is not supported, or will be supported by EF?
As a workaround, I am fetching the entity normally, then doing a map to a DTO. However I would prefer support for projection for performance reasons.
Thanks in advance!
Further technical details
EF Core version: 2.0.0
Database Provider: NpgSql
Operating system: Windows 10 Pro
IDE: Visual Studio 2017 15.5.1

@BenjiZombie We will try to get a fix for this into 2.1, although it might not make it. However, fixing this exception might still result in an expression tree that EF cannot translate. We prioritize expression trees that come from lambda expressions directly over arbitrary trees that can be build manually or by other tools.
You might want to consider submitting a PR to get this fixed sooner.

Thanks for the quick reply. I'll investigate doing a PR if time allows.

See #18685

authors: BenjiZombie ajcvickers smitpatel

repo: aspnet/EntityFrameworkCore