text:
rust_xlsxwriter Roadmap

rust_xlsxwriter is a Rust library that can be used to write text, numbers, dates and formulas to multiple worksheets in an Excel 2007+ XLSX file.
It is a port of the XlsxWriter Python module by the same author.  I also actively maintain a C version libxlsxwriter and a Perl version Excel::Writer::XLSX. The Rust version will also try to address some limitations and frequently requested features of the previous versions, such as the separation of formatting and data writing.
The overall focus of rust_xlsxwriter is on performance, on testing, on documentation, and on fidelity with the file format created by Excel. Reading or modifying files is outside the scope of this library.
New features will be added in approximately this order:
Phase 1: Basic functionality

[x] mdBook documentation - Done: Working with rust_xlsxwriter
[x] Support for worksheets decoupled from workbook to avoid ownership issues with more than one worksheet. Done: Creating Worksheets
[x] Worksheet page setup functions. Done: Page Setup
[x] Freeze panes. Done: set_freeze_panes()
[x] Document properties/metadata. Done
[ ] Worksheet protection Example
[x] URL/Hyperlink support. Done: write_url()
[x] Merge Range support. Done: merge_range()
[x] Simulated autofit. Done: set_autofit()
[x] Tab manipulation. Working with worksheet tabs

Phase 2: Medium functionality

[x] Image support. Done: Image
[x]  Images in headers. Done: Headers and Footers
[ ] URLs in images.
[x] Autofit - simulated: Done
[x] Defined names. Done
[ ] Low memory mode. Example
[x] In memory mode. Done: workbook.save_to_buffer()
[ ] Tempfile directory support. Example
[x] Rich strings. Done
[x] Dynamic functions . Done
[ ] Features/Optional dependencies.
[ ] Format merging

Phase 3: Larger functionality

[x] Autofilters. Partially implemented
[ ] Charts. Example
[ ] Tables. Example
[ ] Conditional formatting. Example
[ ] Data Validation. Example
[ ] Other XlsxWriter features

Initial performance data:
$ hyperfine target/release/examples/app_perf_test ./c_perf_test "python py_perf_test.py"
Benchmark 1: target/release/examples/app_perf_test
  Time (mean ± σ):     447.2 ms ±   8.3 ms    [User: 402.8 ms, System: 39.4 ms]
  Range (min … max):   431.5 ms … 460.7 ms    10 runs

Benchmark 2: ./c_perf_test
  Time (mean ± σ):     362.5 ms ±   6.4 ms    [User: 305.2 ms, System: 53.0 ms]
  Range (min … max):   353.2 ms … 371.9 ms    10 runs

Benchmark 3: python py_perf_test.py
  Time (mean ± σ):      2.899 s ±  0.023 s    [User: 2.787 s, System: 0.088 s]
  Range (min … max):    2.868 s …  2.934 s    10 runs

Summary
  './c_perf_test' ran
    1.23 ± 0.03 times faster than 'target/release/examples/app_perf_test'
    8.00 ± 0.16 times faster than 'python py_perf_test.py'

Or in other words, the C version is the fastest and if we take that as 1 then the rust version is 1.2x (or 20%) slower and the Python version is 8x slower.
The Rust version is ~ 6.5 x faster than the Python version.

$ hyperfine target/release/examples/app_perf_test "python py_perf_test.py"
Benchmark 1: target/release/examples/app_perf_test
  Time (mean ± σ):     450.8 ms ±   5.5 ms    [User: 406.8 ms, System: 39.1 ms]
  Range (min … max):   443.5 ms … 459.1 ms    10 runs

Benchmark 2: python py_perf_test.py
  Time (mean ± σ):      2.942 s ±  0.040 s    [User: 2.821 s, System: 0.090 s]
  Range (min … max):    2.877 s …  3.014 s    10 runs

Summary
  'target/release/examples/app_perf_test' ran
    6.53 ± 0.12 times faster than 'python py_perf_test.py'

Is polars support planned? I saw the python xlsxwriter have pandas support, I wonder if the rust xlsxwriter have any plans to support polars.

I wonder if the rust xlsxwriter have any plans to support polars.

Good suggestion. That was something that I was thinking about. I wrote the initial xlsxwriter integration into Pandas. I'll have a look in their GitHub issues/requests and see if there is any planned work.

Chart support is the next major feature under development. This has a roadmap of it own since it is has a lot of properties and methods:

[ ] Initial support for Bar charts

[ ] Initial support for all other major chart types

[ ] Series option:

[ ] name: Set the name for the series.
[ ] line: Set the properties of the series line type such as color and width.
[ ] border: Set the border properties of the series such as color and style.
[ ] fill: Set the solid fill properties of the series such as color.
[ ] pattern: Set the pattern fill properties of the series.
[ ] gradient: Set the gradient fill properties of the series.
[ ] marker: Set the properties of the series marker such as style and color.
[ ] trendline: Set the properties of the series trendline.
[ ] smooth: Set the smooth property of a line series.
[ ] y_error_bars: Set vertical error bounds for a chart series.
[ ] x_error_bars: Set horizontal error bounds for a chart series.
[ ] data_labels: Set data labels for the series.
[ ] points: Set properties for individual points in a series.
[ ] invert_if_negative: Invert the fill color for negative values.
[ ] overlap: Set the overlap between series in a Bar/Column chart.
[ ] gap: Set the gap between series in a Bar/Column chart.
[ ] chart.set_y_axis()

[ ] chart.set_x2_axis()

[ ] chart.set_y2_axis()

[ ] chart.combine()

[ ] chart.set_size()

[ ] chart.set_title()

[ ] chart.set_legend()

[ ] chart.set_chartarea()

[ ] chart.set_plotarea()

[ ] chart.set_style()

[ ] chart.set_table()

[ ] chart.set_up_down_bars()

[ ] chart.set_drop_lines()

[ ] chart.set_high_low_lines()

[ ] chart.show_blanks_as()

[ ] chart.show_hidden_data()

[ ] chart.set_rotation()

[ ] chart.set_hole_size()

Is it possible to use rayon or std::thread::scope (parallelism) to do each worksheet and at the end add in the final workbook?

Is it possible to use rayon or std:🧵:scope (parallelism) to do each worksheet and at the end add in the final workbook?

It wouldn't be easy. I've thought a good bit about this in the past in relation to the other language version of the library. The main issue is that the xlsx file format has a lot of interlinked "relationships" stored in .rel files. Worksheet strings are also stored in a shared hash table and referenced by id. These, more or less, need to be worked out sequentially.
Not to say that it is impossible, nothing really is in software, just hard and potentially error prone.
Do you have a performance bottleneck or some use case that would benefit from parallelism or is it just a general question?

Just a general question to speed up printing excel files with parallelism.
Thanks!

@claudiofsr that is not a bad idea/implementation. The problem, as you probably figured out, is that threading would be most beneficial at the point where each worksheet is written but that is also the part where the interlocks happen. And again it wouldn't be impossible, just very tricky/not worth the effort.
The main use case where threading would be beneficial is with a medium to large number of large worksheets. However in most use cases people have a medium number of small worksheets or one large worksheet (although of course there are always exceptions). If the most common use case was large number of large worksheets then I would probably try to redesign to facilitate threading but I think the current design is probably the right balance of performance for most people and writability for me.
Finally, just on overall performance the latest version of rust_xlsxwriter is 10% faster that previous versions and also supports a new feature called zlib which uses the C zlib library to boost performance by about 1.6x and is even 1.3x faster than the native C version libxlsxwriter. See #23 (Ideas for performance) and the updated benchmarks at https://rustxlsxwriter.github.io/performance.html

Good morning, jmcnamara.
I just updated rust_xlsxwriter to version 0.34. I tested both libraries: rust_xlsxwriter and xlsxwriter. I noticed that xlsxwriter is twice as fast as rust_xlsxwriter.
Another question. Is it possible to change or add worksheet names without relying on the workbook?

Is it possible to change or add worksheet names without relying on the workbook?

Yes. There is a set_name() method .

I suggest something like this:
let worksheet = Worksheet::new().set_name("Name XYZ")?;
And then:
workbook.push_worksheet(worksheet);

@claudiofsr It is probably better to open a separate question for this.

Correcting my previous text.
I just updated rust_xlsxwriter to version 0.36.1.
I tested both libraries: rust_xlsxwriter and xlsxwriter.
Now, I noticed that xlsxwriter and rust_xlsxwriter are equivalent in terms of processing time!
Congratulations!

I would be really happy to see the tables functionality added

That will be the next major feature after I complete more of the chart feature.

@Christoph-AK see #41 for initial table support.

Hi.. Thanks for the library!! I was in need of an active excel writer library in rust for my new project and this project seems to be most active and promising.
My project may depend on a lot of existing excel templates. I understand from the readme that currently editing an existing excel file is not supported. Any chances that this could be added in the future?

Any chances that this could be added in the future?

Unfortunately no, I don’t plan to tackle reading or rewriting XLSX files.
The  XLSX structure is difficult to parse and rewrite for anything beyond basic data (and even that it can be hard for elements like dates).
Instead I’m going to concentrate my efforts to try give Rust a best in class XLSX writing library.
For additional context here is a reply that I gave to a similar request to the Python version of the library: https://github.com/jmcnamara/XlsxWriter/issues/653#issuecomment-607173901

I've added support for Conditional Formatting. See Working with Conditional Formats in the docs.

I'm looking for some usability feedback for an initial implementation of support for Serde serialization if anyone would like to try serializing data to an Excel worksheet using rust_xlsxwriter. Please comment in the other issue thread (#61).
The code is on main. You will need to enable it in a local project with the following or similar:
cargo add --git https://github.com/jmcnamara/rust_xlsxwriter.git -F serde

I have added support for Serde serialization in v0.57.0. See Working with Serde in the rust_xlsxwriter docs and the discussion thread #61.
Some additional serialisation features and helpers will be added in upcoming releases.

Do you have any plans regarding reaching version 1.0? You seem to make a new 0.x release every week or so. Many of them don't have any breaking changes, but still require me to manually bump the version in Cargo.toml to make sure that I don't miss any new bug fixes. It would be nice to actually utilize semantic versioning and indicate non-breaking releases with 1.x bumps. This way, my app would be able to depend on the major version and automatically receive library updates. Note that I'm not asking for any new stability commitments, you can release 2.0 as soon as you want to make a change. This model would already be more comfortable than a breaking bump on every release.
In theory, I assume that major numbers would also make the life easier for library authors who depend on rust_xlsxwriter. Currently, they have to either pin it to a very narrow minor version or define a range with an explicit upper bound and bump it every week.

Obligatory mention of https://github.com/obi1kenobi/cargo-semver-checks - might be worth including.

Do you have any plans regarding reaching version 1.0?

I plan to release a 1.x.x version once the feature set is ~ 100% of the Python feature set. Based on the task list above the current feature set is 25/33 features and based on ported test cases it is ~ 700/900. I would hope to get to 1.0.0 by the end of the year.

You seem to make a new 0.x release every week or so.

That will probably continue although through this year (with an upcoming pause of 1-2 months while I work on some of the other language libraries/features).

Many of them don't have any breaking changes, but still require me to manually bump the version in Cargo.toml to make sure that I don't miss any new bug fixes.

Yes. Some, or many, of those could be path level releases but most contain a reasonable level of new functionality.
The semver docs say:

Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backward compatible functionality is introduced to the public API. It MUST be incremented if any public API functionality is marked as deprecated. It MAY be incremented if substantial new functionality or improvements are introduced within the private code. It MAY include patch level changes. Patch version MUST be reset to 0 when minor version is incremented.

I am usually in the "MAY" category and sometimes in the "MUST".

Note that I'm not asking for any new stability commitments, you can release 2.0 as soon as you want to make a change. This model would already be more comfortable than a breaking bump on every release.

I think I would end up incrementing a large number of major versions as well. I don't know if that would be better or worse for the end user.
Anyway, overall I think you (and others) will just need to bear with me for the next year or so. The downside is that there will be several more bumps in minor versions but the (hopefully) upside is that there will be new substantive features added on a regular basis.

Obligatory mention of https://github.com/obi1kenobi/cargo-semver-checks - might be worth including.

Thanks. I have tried it in the past and find it a bit hard to use/parse. No disrespect to the devs, it is clearly a good tool. I (personally) just haven't managed to put it to good use. I might try again to at least work out if some releases could be patch releases.

I have released rust_xlsxwriter v0.63.0 with support for embedding images into worksheets. See the Embedded Images example in the docs.
This can be useful if you are building up a spreadsheet of products with a column of images of each product. Embedded images move with the cell so they can be used in worksheet tables or data ranges that will be sorted or filtered.
This functionality is the equivalent of Excel's menu option to insert an image using the option to "Place in Cell" which is available in Excel 365 versions from 2023 onwards. I was a frequently requested feature for Excel and for the xlsxwriter variants.

I have released rust_xlsxwriter v0.64.0 with support for sparklines. See the Working with Sparklines section in the docs.
This is a somewhat niche feature and as far as I can tell not widely used in the Python version. However, it is needed for compatibility with Polars which is currently one of my priorities.

A +1 for low memory mode, in case such feedback is useful :)

A +1 for low memory mode, in case such feedback is useful :)

It is useful. :-)

I've added support for adding Excel data validations to worksheet cells in rust_xlsxwriter v0.70.0.
Data validation is a feature of Excel that allows you to restrict the data that a user enters in a cell and to display associated help and warning messages. It also allows you to restrict input to values in a dropdown list.
Here is an example:
use rust_xlsxwriter::{DataValidation, DataValidationRule, Workbook, XlsxError};

fn main() -> Result<(), XlsxError> {
    // Create a new Excel file object.
    let mut workbook = Workbook::new();
    let worksheet = workbook.add_worksheet();

    worksheet.write(1, 0, "Enter rating in cell D2:")?;

    let data_validation = DataValidation::new()
        .allow_whole_number(DataValidationRule::Between(1, 5))
        .set_input_title("Enter a star rating!")?
        .set_input_message("Enter rating 1-5.\nWhole numbers only.")?
        .set_error_title("Value outside allowed range")?
        .set_error_message("The input value must be an integer in the range 1-5.")?;

    worksheet.add_data_validation(1, 3, 1, 3, &data_validation)?;

    // Save the file.
    workbook.save("data_validation.xlsx")?;

    Ok(())
}

Output:

See DataValidation for details.

I've added support for adding VBA Macros to rust_xlsxwriter using files extracted from Excel files. This isn't very useful and it is also a little kludgy but it is a reasonably popular feature of the Python library and it has utility in some circumstance.
Also, this lays some of the groundwork for adding cell comments (now called Notes by Excel).
Explanation
An Excel xlsm file is structurally the same as an xlsx file except that it contains an additional vbaProject.bin binary file containing VBA functions and/or macros.
Unlike other components of an xlsx/xlsm file this data isn't stored in an XML format. Instead the functions and macros as stored as a pre-parsed binary format. As such it wouldn't be feasible to programmatically define macros and create a vbaProject.bin file from scratch (at least not in the remaining lifespan and interest levels of the author).
Instead, as a workaround, the Rust vba_extract utility is used to extract vbaProject.bin files from existing xlsm files which can then be added to rust_xlsxwriter files.
See Working with VBA Macros.
Here is an example:
use rust_xlsxwriter::{Button, Workbook, XlsxError};

fn main() -> Result<(), XlsxError> {
    // Create a new Excel file object.
    let mut workbook = Workbook::new();

    // Add the VBA macro file.
    workbook.add_vba_project("examples/vbaProject.bin")?;

    // Add a worksheet and some text.
    let worksheet = workbook.add_worksheet();

    // Widen the first column for clarity.
    worksheet.set_column_width(0, 30)?;

    worksheet.write(2, 0, "Press the button to say hello:")?;

    // Add a button tied to a macro in the VBA project.
    let button = Button::new()
        .set_caption("Press Me")
        .set_macro("say_hello")
        .set_width(80)
        .set_height(30);

    worksheet.insert_button(2, 1, &button)?;

    // Save the file to disk. Note the `.xlsm` extension. This is required by
    // Excel or it raise a warning.
    workbook.save("macros.xlsm")?;

    Ok(())
}

Output:

I've dded support for cell Notes (previously called Comments) in v0.72.0.
See https://docs.rs/rust_xlsxwriter/latest/rust_xlsxwriter/struct.Note.html
Here is an example:
use rust_xlsxwriter::{Note, Workbook, XlsxError};

fn main() -> Result<(), XlsxError> {
    // Create a new Excel file object.
    let mut workbook = Workbook::new();

    // Add a worksheet to the workbook.
    let worksheet = workbook.add_worksheet();

    // Widen the first column for clarity.
    worksheet.set_column_width(0, 16)?;

    // Write some data.
    let party_items = [
        "Invitations",
        "Doors",
        "Flowers",
        "Champagne",
        "Menu",
        "Peter",
    ];
    worksheet.write_column(0, 0, party_items)?;

    // Create a new worksheet Note.
    let note = Note::new("I will get the flowers myself").set_author("Clarissa Dalloway");

    // Add the note to a cell.
    worksheet.insert_note(2, 0, &note)?;

    // Save the file to disk.
    workbook.save("notes.xlsx")?;

    Ok(())
}

And the output:

I didn't port some of the Python Note/Comment features such as note positioning since they weren't widely used and Excel's implementation tends to surprise people. If people ask of them I'll add them. The infrastructure is already in place.
Note, in versions of Excel prior to Office 365 Notes were referred to as "Comments". The name Comment is now used for a newer style threaded comment and Note is used for the older non threaded version. The newer Threaded Comments are unlikely to be added to rust_xlsxwriter due to fact that it relies on company specific metadata to identify the comment author.
As an aside the internal traits that I had put in place for other worksheet objects (images, charts, buttons) made this feature relatively easy to add. I really like this aspect of Rust where some of the abstractions can give very clean and easy to maintain/refactor code. Overall I really enjoy using Rust as a language.

I've uploaded version v0.74.0 of rust_xlsxwriter which adds methods to format cells separately from the data writing functions.
In Excel the data in a worksheet cell is comprised of a type, a value and a format. When using rust_xlsxwriter the type is inferred and the value and format are generally written at the same time using methods like Worksheet::write_with_format().
However, if required you can now write the data separately and then add the format using the new methods like Worksheet::set_cell_format(),Worksheet::set_range_format() and Worksheet::set_range_format_with_border().
For example you can now create border formatting like this with a single method call:

This has always been a heavily requested feature in the Python version but due to some different design decisions it was never easy to implement.

I've released v0.75.0 of rust_xlsxwriter which removes  the dependency on the regex.rs crate for smaller binary sizes. The only non-optional dependency is now zip.rs.
An example of the size difference is shown below for one of the sample apps:

app_hello_world
v0.74.0
v0.75.0

Debug
9.2M
4.2M

Release
3.4M
1.6M

See the discussion at #108

Version v0.76.0 of rust_xlsxwriter is out with added support for adding Textbox shapes to worksheets.

See the documentation for Shape.
This is the last of the "Larger functionality" features. We are probably on track to be feature complete with the Python version by the end of the year.

Folks, I am looking for some input on "constant_memory" mode for rust_xlsxwriter: https://github.com/jmcnamara/rust_xlsxwriter/issues/111

Released version v0.77.0 of rust_xlsxwriter with support for Chartsheets.
A Chartsheet in Excel is a specialized type of worksheet that doesn't have cells but instead is used to display a single chart. It supports worksheet display options such as headers and footers, margins, tab selection and print properties.

Chartsheets aren't widely used these days (as far as I can see) but end users sometimes request this feature.

I have an initial working version of the "constant memory" mode on the constant_memory branch (see #111 ). It currently has limited functionality but there is enough to allow me to benchmark the potential savings. I'm reposting the results here for a wider audience and hopefully some feedback/testing.
The memory usage profile is effectively flat (as designed):

Cells
Standard - Size (MB)
Constant Memory - Size (MB)
Standard - Time (s)
Constant Memory - Time (s)

100,000
16.213
0.021
0.101
0.088

200,000
32.405
0.021
0.214
0.179

300,000
52.794
0.021
0.335
0.276

400,000
64.793
0.021
0.443
0.369

500,000
76.792
0.021
0.564
0.468

600,000
105.569
0.021
0.673
0.564

700,000
117.567
0.021
0.768
0.669

800,000
129.567
0.021
0.874
0.799

900,000
141.566
0.021
1.002
0.862

1,000,000
153.565
0.021
1.081
1.022

Which looks like this:

Similarly to the Python version the performance is also slightly better (5-15%) in this mode. Lower time is better. However for numeric only/heavy data (which is the case in practice) the performance is more of less the same.

The tests were run like this:
./target/release/examples/app_memory_test 4000
./target/release/examples/app_memory_test 4000 --constant-memory

So the initial results are good. I'll continue with the functionality.
It would be good to have a few other eyes on this in #111.

How about the serde sub-struct flattening support?

How about the serde sub-struct flattening support?

@hackers267 It is probably worth opening a feature request for that.
The main problem is how would a sub-structure be flattened in the 2D cell matrix of a worksheet. The obvious approach would be to flatten the sub-structure into a a string in a cell, but that might not be a great solution for some (most?) users.
So it is probably best to kick off a Feature Request issue with an example of your data structure and also an Excel example of what you think the output should look like. Anyone else who is facing this issue can then add their opinion/suggestion.

I have an initial working version of the "constant memory" mode on the constant_memory branch (see #111 ).

Wow, this looks incredibly promising. Are there reasons not to use this as default, or maybe switch to it intelligently if only supported features are used?

How about the serde sub-struct flattening support?

@hackers267 It is probably worth opening a feature request for that.
The main problem is how would a sub-structure be flattened in the 2D cell matrix of a worksheet. The obvious approach would be to flatten the sub-structure into a a string in a cell, but that might not be a great solution for some (most?) users.
So it is probably best to kick off a Feature Request issue with an example of your data structure and also an Excel example of what you think the output should look like. Anyone else who is facing this issue can then add their opinion/suggestion.

@jmcnamara  I'm dealing with a scenario where in the generated excel, there are some fixed columns like name, age, gender, etc. and some unfixed columns like 2022 results, 2023 results, January 2024 results... Until results of the month. In this, the column that is fixed can be represented by the field of Struct, while the column that changes can be represented by the sub-struct of the flatten of serde, which can be of HashMap type. I think this is one of the application scenarios for sub-structure.

Wow, this looks incredibly promising. Are there reasons not to use this as default, or maybe switch to it intelligently if only supported features are used?

@Christoph-AK There are a number of restrictions around "constant memory" mode that don't make it suitable for all use case. Such as:

Data needs to be written in strict row/column order. Once row n has been written it is no longer possible to write to any row that is n - 1. This is generally okay when you are converting large datasets that are structured in that way but even then they are often structured based on columns (like Pandas or Polars).
Some Excel functionality like merged ranges, tables and array formulas write across multiple rows in one go and therefore they move the row number forward and don't allow the user to update previous rows. I will fix this in the upstreamed version using a lookahead buffer for the merge ranges and array formula cases. Tables may take a bit longer to resolve since it contains more edges cases.
Embedded (but not inserted) images need to adhere to the row by row order.
The "constant memory" mode works by writing data to disk instead of keeping it in memory and this means that each worksheet in constant memory mode consumes a filehandle. This has been been an issue with users of the Python/C versions who wanted to create workbooks with thousands of worksheets and ran into the ulimit for open files. Not a common occurrence but still one that happened enough for it to be reported.
On systems where the disk is loaded in memory there isn't any memory saving. :-) This has also been reported a few times.

So in general there are a few too many gotchas to turn it on as the default. In fact it will probably be behind a feature flag since it will require an additional dependency on the tempfile crate.
However, for use cases where it makes sense it will be a big efficiency gain. Also, it is possible to turn it on on a worksheet by worksheet basis so users can mix and match worksheets are required.

I'm dealing with a scenario where in the generated excel, there are some fixed columns like name, age, gender, etc. and some unfixed columns like 2022 results, 2023 results, January 2024 results... Until results of the month. In this, the column that is fixed can be represented by the field of Struct, while the column that changes can be represented by the sub-struct of the flatten of serde, which can be of HashMap type

@hackers267 Thanks for the description. The HashMap type at least may be feasible to support. Open a feature request and put in a sample struct and some example data and I will look at what can be done.

Released v0.78.0 with support for "constant memory" mode to reduce memory usage when writing large worksheets.
The constant_memory mode works by flushing the current row of data to disk when the user writes to a new row of data. This limits the overhead to one row of data stored in memory. Once this happens it is no longer possible to write to a previous row since the data in the Excel file must be in row order. As such this imposes the limitation of having to structure your code to write in row by row order. The benefit is that the required memory usage is very low,and effectively constant, regardless of the amount of data written.

@jpmckinney You asked about this a while back. If you are still interested you can check it out.

I've released v0.79.0 of rust_xlsxwriter with support for files larger than 4GB.
The rust_xlsxwriter library uses the zip.rs crate to provide the zip container for the xlsx file that it generates. The size limit for a standard zip file is 4GB for the overall container or for any of the uncompressed files within it.  Anything greater than that requires ZIP64 support. In practice this would apply to worksheets with approximately 150 million cells, or more.
See Workbook::use_zip_large_file().

For anyone who is interested I wanted to highlight a new wasm wrapper for rust_xlsxwriter that looks fairly complete even in the initial stages: https://github.com/estie-inc/wasm-xlsxwriter

Hi @jmcnamara,
I currently use nodejs/exceljs to write very large xlsx files. I'm looking into potentially faster alternatives, including this lib.
I have created a quick benchmark comparing different language/lib combinations  https://github.com/brunoargolo/excel-writing-multilanguage-benchmark
I'm a 100% beginner in Rust, GO and Python. So I might have royally fudged something.
I was hoping Rust would prove significantly faster than node but its pretty much a tie.
If you find any mistakes in my approach please let me know, I tried to make the repo as easy to reproduce as possible so anyone can test it.
Node and Go are using the streaming function of their respective libs, on python I used the constant memory mode on rust I tried both low_memory and constant memory with similar results.

Language + Lib
1 sheet
9 sheets

Nodejs with exceljs
5s
41s

Go with excelize
5s
30s

Python3 with xlsxwriter
26s
236s

Rust with rust_xlsxwriter
5s
40s

I tried to make the repo as easy to reproduce as possible so anyone can test it.

Thanks. That is helpful.
Overall, I don't see anything wrong with the benchmarks although the rust version does a bit too much type conversion. A real world example would probably use serde_json, or similar, and do the type conversion as part of the read step. However, this adds at most a 10% overhead to the write phase. Also, in this particular case the add_worksheet_with_constant_memory() method should be 5-10% faster.
I also crosschecked with the examples/app_perf_test.rs program for a similar number of cells and the runtime was approximately the same.
So overall, with some fine tuning and you could get 1.1-1.2 times better performance but that would still be in the same ballpark of the nodejs/exceljs solution.

I really appreciate you taking the time to go over this, thank you!
My guess is we're probably hitting the limits of what is viable in any language... I imagine we're not going to see 2x or 3x faster no matter the language/lib combination.
The one thing that might be significantly faster could be writing to XLSB format, but it is probably a ridiculous amount of work...
Do you still have any major performance related items on the roadmap or do you think this is mostly as good as it gets plus or minus a few percentage points?

My guess is we're probably hitting the limits of what is viable in any language... I imagine we're not going to see 2x or 3x faster no matter the language/lib combination.

Agreed. The bottlenecks come down to IO/writing the worksheet.xml files and zipping them and other supporting files into a zip container. So most implementations will probably hit the same performance floor. Notwithstanding that I'm impressed with the ExcelJS performance.

Do you still have any major performance related items on the roadmap or do you think this is mostly as good as it gets plus or minus a few percentage points?

There may be some small (< 10%) IO/writing gains still to be made but most of the low hanging fruit have  been picked. A potentially bigger gain would probably come from a threaded/parallelized zip library.
The C library gets a 20-30% performance improvement from using the Milo Yip DTOA library library for float to string conversion but I didn't get a similar speed increase from the ryu library except in very large files.
It is also possible that a properly parallelized backend worksheet assembly using rayon or similar might improve multiple worksheet writing. I have thread::scope parallelization in the backend but someone with more multithreaded rust experience might be able to do better:
https://github.com/jmcnamara/rust_xlsxwriter/blob/main/src/packager.rs#L122-L135
But to take a step back it is worth noting that the Rust version is at almost 90% of the C library performance so any 2-3x gain is unlikely without a radically different approach.

The one thing that might be significantly faster could be writing to XLSB format, but it is probably a ridiculous amount of work...

Yes. I maintained a library for the similar XLS format and it wasn't fun to maintain/extend. So for me that would be a non-runner. :-)

I wonder if a different allocator can give more performance in this benchmark.
Combining all approaches (allocator, faster serialisation and floats, better parallelization) would probably approach the go time.
I think garbage collected languages like go and js have an advantage vs default rust here as the gc can easier reuse allocated memory, and go does have an advantage when it comes to easily throwing multithreading at everything.
Of cause all of those characteristics can be programmed in rust with some care, so there shouldn't really be a language other then c thats inherently faster.

I've released v0.80.0 or rust_xlsxwriter with an improvement/fix for heap memory usage in constant_memory mode. This is a recommended upgrade for anyone using that mode/feature. See #120 for more information.

authors: Christoph-AK Expurple Plus-1xp brunoargolo claudiofsr hackers267 jmcnamara jpmckinney pickfire


text:
Proposal: generic write() method

It would be possible to have a generic write() method using a trait that could call the type specific write_*() method such as  write_number(). This is a feature in the Python version of the library.
I'll leave this here as a reminder to myself and to see if it gets an +1s for usefulness or desirability.

Can I take a shot at this?

@moutikabdessabour At this stage I'm mainly asking if people think this would be useful. Do you think it would be useful?
Personally I'm currently more against this than for the reasons below but I'm willing to take input.
Pros:

Makes writing examples easier.

Cons:

The similar write() method in the Python and Perl versions tend to generate bug reports when users try to write arbitrary sting or number like data. People also assume that writing something like "2003-01-01" should become a date in Excel.
There would be a slight performance overhead for strings since they would need to be checked to see if they are formulas or urls

For the second con I think implementing a Formula/Url type would remove the overhead. I wrote an app using this library and while it is great the only issue I had is the verbosity of the function calls. I am a 100% for this feature

I wrote an app using this library and while it is great the only issue I had is the verbosity of the function calls.

Yes, that is my feeling too. There are currently 24 write_*() methods. That is just too many.
It could be mapped down to 5 like this:

Current
New

write_number()
write()

write_string()
write()

write_url()
write()

write_boolean()
write()

write_formula()
write()

write_rich_string()
write()

write_blank()
write()

write_time()
write()

write_date()
write()

write_datetime()
write()

write_dynamic_formula()
write()

write_number_with_format()
write_with_format()

write_string_with_format()
write_with_format()

write_url_with_format()
write_with_format()

write_boolean_with_format()
write_with_format()

write_formula_with_format()
write_with_format()

write_rich_string_with_format()
write_with_format()

write_dynamic_formula_with_format()
write_with_format()

write_array_formula()
write_array_formula()

write_dynamic_array_formula()
write_array_formula()

write_array_formula_with_format()
write_array_formula_with_format()

write_dynamic_array_formula_with_format()
write_array_formula_with_format()

write_url_with_options()
Same

write_url_with_text()
Same, or write_url_with_options()

With some caveats/additions:

It will need the addition of a new Url type.
It will need the addition of a new Formula type that can be set to array or dynamic array (if needed)
Default DateTime, Date and Time formats for datetime()/date()/time() variants without a user specified format since a date without a format in Excel is just a number.

Also a IntoExcelType (or similar) trait coupled with a generic write() would have the advantage of allowing mapping of other arbitrary user types in a clean way.
Now, that I've mapped it out like this it seems like the right thing to do (even if it is another major API break). I might add it after I finish the current work on charts.

Gonna take a jab at this later today. WIll probably have something ready tomorrow night.

Not to deter you from doing a prototype. I just to flag that it will needs a lot of doc and example refactoring and I'd prefer to do that myself.

i would love the feature as it would enable me for example to write one function to write json data in array format (structured differently) with having to write many functions for a specific struct to write json to excel. If you dim this feature as unnecessary then can you link me a rs lib that enable me todo so if its available.

@GlennMm At this point I'm definitely going to add it. Either in the next release or a separate release just after.

Also, if the Format was treated as an Option it would be possible to have just one method that needs to be implemented for the trait:
impl IntoExcelData for UnixTime {
    fn write<'a>(
        self,
        worksheet: &'a mut Worksheet,
        row: RowNum,
        col: ColNum,
        format: Option<&'a Format>,
    ) -> Result<&'a mut Worksheet, XlsxError> {
        // Convert the Unix time to an Excel datetime.
        let datetime = 25569.0 + (self.seconds / (24 * 60 * 60)) as f64;

        match format {
            Some(format) => {
                // Write the date with the user supplied format.
                worksheet.write_number_with_format(row, col, datetime, format)
            }
            None => {
                // Create a default date format.
                let format = Format::new().set_num_format("yyyy-mm-dd");
                worksheet.write_number_with_format(row, col, datetime, &format)
            }
        }
    }
}

This may, or may not, be more confusing to the end user. Internally it works out a lot cleaner.

@Fight-For-Food Any thoughts on the interface or implementation of this feature? You had some good input on other generic/non-generic code.

@Fight-For-Food Any thoughts on the interface or implementation of this feature? You had some good input on other generic/non-generic code.

I think that
fn write(self, worksheet: &mut Worksheet, row: RowNum, col: ColNum) -> Result<&mut Worksheet, XlsxError>{....}
is ok. It is convenient to use. And there is no need to make in more complex with extra-argument format: Option<&'a Format>
For some more comprehensive thoughts i need more time. Maybe i will have some at this week. But can't make a promise about it.

And there is no need to make in more complex with extra-argument format: Option<&'a Format>

It may not be clear from the snippet above but the addition of the Option<> means that there is only one trait funtion that needs to be implemented by the end user. However, it would probably be confusing and the version on main with IntoExcelData::write() and IntoExcelData::write_with_format() mimics the worksheet APIs may be simpler.

Maybe i will have some at this week. But can't make a promise about it.

No problem if you can't get to it.

thank you for implementing this, it really help make my work much easier. looking to forward to be start contributing to this repo. so have started using this feature i haven't seen any problems with it so far

Just a heads up that the initial version of this is upstream in crates.io in version 0.27.0: https://crates.io/crates/rust_xlsxwriter
I'll iterate and improve on this in the next few releases.

Having taken a small detour into charts I've come back to complete this.
Some tasks, not in order:

Use impl Into<String> instead of &str in method parameters where possible/sensible.
Implement generic write_row() and write_column() methods.
Add types for Url and Formula and add them to generic write().

Implement generic write_row() and write_column() methods.

I've added a first pass at this to main. You can now write arrays of data types that implement IntoExcelData like this:
use rust_xlsxwriter::{Workbook, XlsxError};

fn main() -> Result<(), XlsxError> {
    // Create a new Excel file object.
    let mut workbook = Workbook::new();

    // Add a worksheet to the workbook.
    let worksheet = workbook.add_worksheet();

    // Some array data to write.
    let numbers = [1, 2, 3, 4, 5];
    let words = ["Hello"; 5];

    worksheet.write_column(0, 0, &numbers)?;
    worksheet.write_column(0, 1, &words)?;

    worksheet.write_row(0, 3, &numbers)?;
    worksheet.write_row(1, 3, &words)?;

    // Save the file to disk.
    workbook.save("arrays.xlsx")?;

    Ok(())
}

Output:

It is just some syntactic sugar around a loop but it is somewhat popular in the Python version.
It currently has an additional deference/copy and I'm not sure if that can be avoided. Feedback welcome on that or anything else.

I gave some feedback on the last commit. I think that feedback applies on more parts of the codebase.

@adriandelgado

I think the Copy trait bound could be avoided by using the IntoIterator trait.

I initially tried, and failed, to get that working. :-( With my morning brain and 2 cups of coffee I managed to understand/fix it. You can have a look.

I gave some feedback on the last commit. I think that feedback applies on more parts of the codebase.

I think I understood those comments but maybe not so let's check. What I think you are saying is that the API/library should avoid taking any references to data that will then be cloned and should instead use interfaces like Into<String> or IntoInterator. Is that correct or am I missing something else?

I've also added write_row_matrix() and write_column_matrix() methods for 2D arrays/iterators of data.

I think I understood those comments but maybe not so let's check. What I think you are saying is that the API/library should avoid taking any references to data that will then be cloned and should instead use interfaces like Into<String> or IntoInterator. Is that correct or am I missing something else?

That's right. The caller of the function has the responsability to clone if needed. Sometimes the caller already has owned data and knows it wont need it anymore, no clone needed in that case.

I've implemented a Formula type so that it can be used in the generic write():
    worksheet.write(0, 0, Formula::new("=1+2+3"))?;
    worksheet.write(1, 0, Formula::new("=SIN(PI())"))?;

However, I'm not sure how much of a help this is yet.

I had hoped to be able to use Trait Objects and the newer array writing methods to simulate something like writing a heterogeneous vec of vectors like a dataframe.
Something like this:
    let data: Vec<Vec<Box<dyn IntoExcelData>>> = vec![
        vec![
            Box::new("East"),
            Box::new("West"),
            Box::new("North"),
            Box::new("South"),
        ],
        vec![
            Box::new(40),
            Box::new(40),
            Box::new(20),
            Box::new(30)],
        vec![
            Box::new(30),
            Box::new(30),
            Box::new(50),
            Box::new(30)],
        vec![
            Box::new(Formula::new("=B1+C1")),
            Box::new(Formula::new("=B2+C2")),
            Box::new(Formula::new("=B3+C3")),
            Box::new(Formula::new("=B4+C4")),
        ],
    ];

However, I've failed in every attempt so far. Examples like this with a simpler data structure:
use rust_xlsxwriter::{Formula, IntoExcelData, Workbook, XlsxError};

fn main() -> Result<(), XlsxError> {
    // Create a new Excel file object.
    let mut workbook = Workbook::new();

    // Add a worksheet to the workbook.
    let worksheet = workbook.add_worksheet();

    let data: Vec<Box<dyn IntoExcelData>> = vec![
        Box::new("East"),
        Box::new(50),
        Box::new(30),
        Box::new(Formula::new("=B1+C1")),
    ];

    /* The following doesn't work.

        --> examples/app_trait_array.rs:39:9
        |
     40 |         item.write(worksheet, row, 1);
        |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the size of `dyn IntoExcelData`
                                                cannot be statically determined

    */
    let mut row = 1;
    for item in data {
        item.write(worksheet, row, 1);
        row += 1;
    }

    // Save the file to disk.
    workbook.save("arrays.xlsx")?;

    Ok(())
}

Or this:
    let mut row = 1;
    for item in data {
        worksheet.write(row, 0, item)?;
        row += 1;
    }

Which gives errors like this:
   --> examples/app_trait_array3.rs:28:33
    |
28  |         worksheet.write(row, 0, item)?;
    |                   -----         ^^^^ the trait `IntoExcelData` is not implemented
                                          for `Box<dyn IntoExcelData>`
    |                   |
    |                   required by a bound introduced by this call
    |
    = help: the following other types implement trait `IntoExcelData`:

Anyone see a way that trait objects for the IntoExcelData trait might be handled?

See also this example of writing a Polars dataframe to Excel using rust_xlsxwriter: #39

These generic methods and types are now available in release v0.39.0.

source: gharchive/issue

authors: Fight-For-Food GlennMm adriandelgado jmcnamara moutikabdessabour

license_type: permissive

license_source: github-api


text:
Always tries to update de_fayard_buildsrcversions_gradle_plugin

Every time I run ./gradlew buildSrcVersions it says I have to "update" your lib. But as you can see:

I was already using the latest version.

@GuilhE Can you show me what you have in the plugins { ...}  block in build.gradle(.kts)?

@jmfayard:
plugins {
    `kotlin-dsl`
}
repositories {
    mavenCentral()
}

@GuilhE sorry, this buildSrc/build.gradle.kts, I mean ./build.gradle.
But even without that information, my guess that updating the plugin version does nothing, because it's not used by your build. The task always regerenate the version currently used by your build.
There isn't a great way yet to configure plugin versions inside the buildSrc.
I will release this as part of version 0.6.2, I will ping you when it's ready.

Oh I get it:
plugins{
    id "de.fayard.buildSrcVersions" version "0.4.2"
}

Yes this version was unchanged, makes sense the behaviour.
Sure, ping me, I'm glad I can help, and thanks for this great plugin!

@GuilhE
v0.6.2 is released https://github.com/jmfayard/buildSrcVersions/releases/tag/v0.6.2
Have a look at the new :refreshVersions files and how it can be leveraged to configure plugin versions (if you upgrade to Gradle 5.6.2)
https://github.com/jmfayard/buildSrcVersions/issues/77

Hello @jmfayard
Well actually using gradle.properties over Libs it's a step back for me, auto-complete it's important. I prefer :buildSrcVersions intead of :refreshVersions.
I haven't noticed by the time but this is the solution I was searching, I have to migrate my build.gradle to .kts though.

@GuilhE gradle.properties does not replace Libs.kt, it replaces Versions.kt
You can use at the same time :refreshVersions to update the versions and :buildSrcVersions to generate Libs.kt and use auto-complete everywhere.
You can also use only :buildSrcVersions if you prefer. Versions.kt  will still be generated unless plugin.de.fayard.buildSrcVersions  is found in gradle.properties
Beware  that v0.6.2 is broken, better use v0.6.3

@jmfayard ok I had the wrong impression because I was searching for a implementation(Libs.something) in your sample-kotlin and it was missing. I've also took a look in your blog  and it wasn't clear for me. My bad ^^

Or you can also use only :buildSrcVersions if you prefer. Versions.kt will still be generated unless plugin.de.fayard.buildSrcVersions is found in gradle.properties

I'll stick with this solution for now, but I'll also try the :refreshVersions

Beware that v0.6.2 is broken, better use v0.6.3

Yes I was following the issue open by the other user, I had the same problem but I was in the middle of the Groovy to Kotlin translation so I didn't pay attention. v0.6.3 fixes the issue :)

@GuilhE well I have the goal to be clear, so if you didn't understand it, it's also "my bad". Is that clearer like this? https://github.com/jmfayard/buildSrcVersions/issues/77

@jmfayard it's perfect! :)

authors: GuilhE jmfayard

repo: jmfayard/buildSrcVersions

license: MIT


text:
JavaReflectionImposteriser and Scala

Hi,
I've been writing a small integration library between JMock and specs2, since we love JMock here at Wix, but we write mostly in Scala now.
I encountered an issue where the JavaReflectionImposteriser would fail to create a mock from a Scala trait that contains a function that returns a Scala Set.
It fails because the Imposteriser uses JDKs Proxy which is unable to create the instance. This is due to Set having two apply methods:
apply(T): Boolean

and
apply(T*:): CC[T]

Since Java ignores Scala's way of creating parameter lists, it things both methods have the same signature but with different return types.
I have worked around this by creating a new Imposteriser, which uses the JavaReflectionImposteriser, and if it fails because of that error, uses the ClassImposteriser instead.
I just thought you guys might find this interesting.

Thanks for the tip. If I understand you correctly you're not raising an issue with JMock?
It looks like we should add your Specs2-JMock project. I nearly always use the ClassImposteriser.

You're correct, this is more of an issue with Scala-Java interop, but I just thought it had some bearing here, perhaps for documentation purposes.
I do like that the JavaReflectionImposteriser is the default and that using the ClassImposteriser has to be a conscious decision, since it doesn't work on Classes, and reminds me that I should strive to use mocking on interfaces, rather than concrete classes.
And that would be great!

authors: nargov olibye

repo: jmock-developers/jmock-library

license_source: bigquery


text:
🛑 UTN Aula Virtual is down

In 0b031f9, UTN Aula Virtual (https://aulavirtual.frbb.utn.edu.ar/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: UTN Aula Virtual is back up in b34fc71 after 44 minutes.

repo: joacoserra/statusPage


text:
🛑 Felix Telecom is down

In f410eec, Felix Telecom (https://www.felixtelecom.ro) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Felix Telecom is back up in 163b036.


text:
🛑 Security & Telecom Systems is down

In 4f93798, Security & Telecom Systems (https://security-telecom.ro) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Security & Telecom Systems is back up in 2880f72 after 14 minutes.


text:
🛑 Security & Telecom Systems is down

In e1288ca, Security & Telecom Systems (https://security-telecom.ro) was down:

HTTP code: 403
Response time: 1900 ms

Resolved: Security & Telecom Systems is back up in f59d489 after 8 minutes.


text:
Added optional manager_agent parameter

As talked about in #466 this adds an optional parameter manager_agent in for the Crew class. It takes in an Agent object that will be used instead of the default one, allowing more granular prompt engineering and even things like scoped delegation.
Things to consider

Documentation?
Check for allow_delegation / tools?

currently it allows delegation to be False / there not to be any delegation tools, which would break "crew" functionality

Could be done easier

user has to make a whole agent
one could also add e.g. manager_agent_role, manager_agent_goal and manager_agent_backstory

I like this! def useful, I'll merge and add some tests next :)

source: gharchive/pull-request

authors: joaomdmoura noggynoggy

repo: joaomdmoura/crewAI


text:
Update Sequential.md

To Resolve :
pydantic_core._pydantic_core.ValidationError: 1 validation error for Task expected_output
Field required [type=missing, input_value=, input_type=dict] For further information visit https://errors.pydantic.dev/2.6/v/missing
"Expected Output" is mandatory now as it forces people to be specific about the expected result and get better result
refer : https://github.com/joaomdmoura/crewAI/issues/308

Great catch!! Merging it!

Great catch!! Merging it!

authors: joaomdmoura stsfaroz


text:
docker configuration

Hi
I run asciidoctor thru docker like this : https://github.com/asciidoctor/docker-asciidoctor
I wonder how configure AsciiDoc.html_generator in vscode to use the docker run instead the default  asciidoctor -o-
Did you ever experiment this ?

You  no longer need asciidoctor to use the extenstion,  Please open the issue if needed. Thanks

authors: Arnotjevleesch joaompinto

repo: joaompinto/asciidoctor-vscode

license: mit


text:
Create DR13 APOGEE-RC sample

Basic commands:
python make_rcsample.py -o ~/tmp/apogee-rc-DR13-addl.fits --rmdups --nostat --nopm --addl-logg-cut

python check_rc_against_apokasc.py ~/tmp/apogee-rc-DR13-addl.fits

Need to do the following:

[x] Check contamination compared to DR12
[ ] Cross-match against DR12 to see how different they are (same input data!)
[x] Decide on whether to apply addl-logg-cut
[x] Cross-match against Tycho
[ ] Make plots from RC paper to see how they've changed
[x] Work on datamodel
[ ] Check and update documentation

Using checks wrt APOKASC:
DR13 catalog (no addl-logg-cut)

19,291 stars
7% contamination --> 5% adjusted
44% completeness

DR13 catalog (w addl-logg-cut)

18,417 stars
5% contamination --> 3.5% adjusted
43% completeness

DR12 catalog (w addl-logg-cut)

19,937 stars
14% contamination (!!) --> 10% adjusted
31% completeness

Conclusion: Don't use ADDL_LOGG_CUT by default

Comparison to DR12

14,288 stars in common: ~5,000 different --> even at ~15% total contamination, seems high.
Metallicity has shifted down by ~0.1 dex, that looks correct based on previous calibration offset
[alpha/Fe] has shifted down by -0.03 dex, exactly what M67's offset was before!
Distance distribution almost exactly the same (off in the median by 2 pc!); l and b also the same

repo: jobovy/apogee


text:
Error trying to run vanilla code: ValueError ("Shapes (?,) and (?,?) must have same rank")

Hi, thanks so much for releasing this code, it's already helped a bunch in helping me go from very basic MNIST models to something more sophisticated.
I'm trying to run your code now but it dies when trying to compile the model. The nvidia card I'm using has very little memory at this point, do you think that's the actual error? The simpler auto-encoder model for the MNIST dataset from http://blog.keras.io/building-autoencoders-in-keras.html works, even when I substitute Adam.
From my reading of the trace, there's the main error, but that's ultimately coming off the dice_coeff functions, though I don't really understand why they would.
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GT 755M
major: 3 minor: 0 memoryClockRate (GHz) 1.0195
pciBusID 0000:01:00.0
Total memory: 2.00GiB
Free memory: 1.65GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:806] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 755M, pci bus id: 0000:01:00.0)

Loading and preprocessing train data...

Creating and compiling model...

ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/tensor_shape.py in merge_with(self, other)
562       try:
--> 563         self.assert_same_rank(other)
564         new_dims = []
/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/tensor_shape.py in assert_same_rank(self, other)
608         raise ValueError(
--> 609             "Shapes %s and %s must have the same rank" % (self, other))
610
ValueError: Shapes (?,) and (?, ?) must have the same rank
During handling of the above exception, another exception occurred:
ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/tensor_shape.py in with_rank(self, rank)
638     try:
--> 639       return self.merge_with(unknown_shape(ndims=rank))
640     except ValueError:
/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/tensor_shape.py in merge_with(self, other)
569         raise ValueError("Shapes %s and %s are not compatible" %
--> 570                          (self, other))
571
ValueError: Shapes (?,) and (?, ?) are not compatible
During handling of the above exception, another exception occurred:
ValueError                                Traceback (most recent call last)
/notebooks/ultrasound-nerve-segmentation/train.py in ()
134
135 if name == 'main':
--> 136     train_and_predict()
/notebooks/ultrasound-nerve-segmentation/train.py in train_and_predict()
102     print('Creating and compiling model...')
103     print('-'*30)
--> 104     model = get_unet()
105     model_checkpoint = ModelCheckpoint('unet.hdf5', monitor='loss', save_best_only=True)
106
/notebooks/ultrasound-nerve-segmentation/train.py in get_unet()
68     model = Model(input=inputs, output=conv10)
69
---> 70     model.compile(optimizer=Adam(lr=1e-5), loss=dice_coef_loss, metrics=[dice_coef])
71
72     return model
/usr/local/lib/python3.4/dist-packages/keras/engine/training.py in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, **kwargs)
587             loss_weight = loss_weights_list[i]
588             output_loss = weighted_loss(y_true, y_pred,
--> 589                                         sample_weight, mask)
590             if len(self.outputs) > 1:
591                 self.metrics.append(output_loss)
/usr/local/lib/python3.4/dist-packages/keras/engine/training.py in weighted(y_true, y_pred, weights, mask)
309     def weighted(y_true, y_pred, weights, mask=None):
310         # score_array has ndim >= 2
--> 311         score_array = fn(y_true, y_pred)
312         if mask is not None:
313             # Cast the mask to floatX to avoid float64 upcasting in theano
/notebooks/ultrasound-nerve-segmentation/train.py in dice_coef_loss(y_true, y_pred)
24
25 def dice_coef_loss(y_true, y_pred):
---> 26     return -dice_coef(y_true, y_pred)
27
28
/notebooks/ultrasound-nerve-segmentation/train.py in dice_coef(y_true, y_pred)
20     y_true_f = K.flatten(y_true)
21     y_pred_f = K.flatten(y_pred)
---> 22     return (2. * K.dot(y_true_f, K.transpose(y_pred_f)) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)
23
24
/usr/local/lib/python3.4/dist-packages/keras/backend/tensorflow_backend.py in dot(x, y)
212         yt = tf.reshape(tf.transpose(y, perm=y_permute_dim), [y_shape[-2], -1])
213         return tf.reshape(tf.matmul(xt, yt), x_shape[:-1] + y_shape[:-2] + y_shape[-1:])
--> 214     out = tf.matmul(x, y)
215     return out
216
/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/math_ops.py in matmul(a, b, transpose_a, transpose_b, a_is_sparse, b_is_sparse, name)
1207                                    transpose_a=transpose_a,
1208                                    transpose_b=transpose_b,
-> 1209                                    name=name)
1210
1211 sparse_matmul = gen_math_ops._sparse_mat_mul
/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/gen_math_ops.py in _mat_mul(a, b, transpose_a, transpose_b, name)
1176   """
1177   result = _op_def_lib.apply_op("MatMul", a=a, b=b, transpose_a=transpose_a,
-> 1178                                 transpose_b=transpose_b, name=name)
1179   return result
1180
/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/op_def_library.py in apply_op(self, op_type_name, name, **keywords)
702           op = g.create_op(op_type_name, inputs, output_types, name=scope,
703                            input_types=input_types, attrs=attr_protos,
--> 704                            op_def=op_def)
705           outputs = op.outputs
706           return _Restructure(ops.convert_n_to_tensor(outputs),
/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py in create_op(self, op_type, inputs, dtypes, input_types, name, attrs, op_def, compute_shapes, compute_device)
2260                     original_op=self._default_original_op, op_def=op_def)
2261     if compute_shapes:
-> 2262       set_shapes_for_outputs(ret)
2263     self._add_op(ret)
2264     self._record_op_seen_by_control_dependencies(ret)
/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py in set_shapes_for_outputs(op)
1700       raise RuntimeError("No shape function registered for standard op: %s"
1701                          % op.type)
-> 1702   shapes = shape_func(op)
1703   if shapes is None:
1704     raise RuntimeError(
/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/common_shapes.py in matmul_shape(op)
84 def matmul_shape(op):
85   """Shape function for a MatMul op."""
---> 86   a_shape = op.inputs[0].get_shape().with_rank(2)
87   transpose_a = op.get_attr("transpose_a")
88   b_shape = op.inputs[1].get_shape().with_rank(2)
/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/tensor_shape.py in with_rank(self, rank)
639       return self.merge_with(unknown_shape(ndims=rank))
640     except ValueError:
--> 641       raise ValueError("Shape %s must have rank %d" % (self, rank))
642
643   def with_rank_at_least(self, rank):
ValueError: Shape (?,) must have rank 2

Hi Thomas, thanks for reporting this - it seems to be the issue when running the code with TF backend (I'm still using Theano). Will check it out and get back to you.

e1433994f38cbaeb44e261169bc5dc6885474787 should fix it. You can pull the repo now and it should work now.

authors: jocicmarko thomas-keller


text:
window is not defined render server side

Hi I implementing you library with react and nextjs, and when I reload the page, I get the error:
window is not defined
ReferenceError: window is not defined
at Object. (/Users/albertocano/Documents/projects/project/project/node_modules/jodit/build/jodit.min.js:9:224)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Module.require (internal/modules/cjs/loader.js:637:17)
at require (internal/modules/cjs/helpers.js:22:18)
at Object.jodit (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:581:18)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at Module../pages/test2.jsx (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:421:63)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at Object.4 (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:503:18)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at /Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:91:18
at Object. (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:94:10)
It's obviously that window obejct is only defined on browser side, but do you can support render server side? please

Same issue.
Is there any work around found to fix this?

For next js you can use this:
import React from 'react';
import dynamic from 'next/dynamic';

const importJodit = () => import('jodit-react');

const JoditEditor = dynamic(importJodit, {
    ssr: false,
});

Hi I implementing you library with react and nextjs, and when I reload the page, I get the error:
window is not defined
ReferenceError: window is not defined
at Object. (/Users/albertocano/Documents/projects/project/project/node_modules/jodit/build/jodit.min.js:9:224)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Module.require (internal/modules/cjs/loader.js:637:17)
at require (internal/modules/cjs/helpers.js:22:18)
at Object.jodit (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:581:18)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at Module../pages/test2.jsx (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:421:63)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at Object.4 (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:503:18)
at webpack_require (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:23:31)
at /Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:91:18
at Object. (/Users/albertocano/Documents/projects/project/project/.next/server/static/development/pages/test2.js:94:10)
It's obviously that window obejct is only defined on browser side, but do you can support render server side? please

For next js you can use this:
import React from 'react';
import dynamic from 'next/dynamic';

const importJodit = () => import('jodit-react');

const JoditEditor = dynamic(importJodit, {
    ssr: false,
});

For Gatsby:
import React from "react";

const Jodit = React.lazy(() => {
  return import("jodit-react");
});

export const RichTextEditor = (props) => {
  const { value, onChange, ...otherProps } = props;
  const isSSR = typeof window === "undefined";

  return (
    <>
      {!isSSR && (
        <React.Suspense fallback={
Loading RTE
}>
          <Jodit value={value} onChange={onChange} {...otherProps}/>
        </React.Suspense>
      )}
    </>
  );
});

For Gatsby:
import React, { useState } from "react";

const Jodit = React.lazy(() => {
  return import("jodit-react");
});

export const RichTextEditor = () => {
  const [value, setValue] = useState("");
  const isSSR = typeof window === "undefined";

  return (
    <>
      {!isSSR && (
        <React.Suspense fallback={
Loading RTE
}>
          <Jodit value={value} onChange={(val) => setValue(val)}/>
        </React.Suspense>
      )}
    </>
  );
});

That solution worked for me @Fedeorlandau, you definitely have a place in heaven!
"gatsby": "^2.18.12"

For Gatsby:
import React, { useState } from "react";

const Jodit = React.lazy(() => {
  return import("jodit-react");
});

export const RichTextEditor = () => {
  const [value, setValue] = useState("");
  const isSSR = typeof window === "undefined";

  return (
    <>
      {!isSSR && (
        <React.Suspense fallback={
Loading RTE
}>
          <Jodit value={value} onChange={(val) => setValue(val)}/>
        </React.Suspense>
      )}
    </>
  );
});

That solution worked for me @Fedeorlandau, you definitely have a place in heaven!
"gatsby": "^2.18.12"

Thank you @oqnunes. I strongly recommend you checking out https://github.com/ianstormtaylor/slate and https://docs.slatejs.org/ before focusing on jodit. I've found an enormous amount of bugs on jodit while working with react and SSR. Some stuff like dynamic configuration for the rte or customize some inputs on the controls aren't 100% possible with jodit. Regards!

authors: Fedeorlandau Gladiolys canotech mahip-nickelfox oqnunes

repo: jodit/jodit-react


text:
Test different image comparisons methods for the image metric

Image comparison

https://github.com/mapbox/pixelmatch
http://rembrandtjs.com/
https://github.com/rsmbl/Resemble.js
https://github.com/bitlyfied/js-image-similarity
https://github.com/HumbleSoftware/js-imagediff/

https://github.com/andrewekhalel/sewar
https://sumn2u.medium.com/quality-metrics-in-image-compression-f6de68df1b8

Did some tests in 4e188b3fd1eb7a4983850436b8967fa0dd883464

Will use https://github.com/andrewekhalel/sewar as a reference implementation in python. And https://github.com/darosh/image-mse-js as something to build on for js code.

Current js-tests available here:
https://joedf.github.io/ImgBeamer/tests/image_metrics/js-tests

Mertric calculation likely needs to update at when a full frame / last row is drawn...

Closing as more or less complete and relatively stable. Open a new issue, if further changes wanted/needed.

repo: joedf/ImgBeamer


text:
small improvement of CONTRIBUTING.md

:+1:

authors: PascalSchumacher joel-costigliola

repo: joel-costigliola/assertj-core


text:
equals() methods generated by jsonschema2pojo trigger warnings in sonarqube

The equals() methods generated are of the general form
@Override
public boolean equals(Object other) {
    if (other == this) {
        return true;
    }
    if ((other instanceof ClassName) == false) {
        return false;
    }
    ClassName rhs = ((ClassName) other);
    return new EqualsBuilder().append(field, rhs.field).append(additionalProperties, rhs.additionalProperties).isEquals();
}

This triggers warnings in sonarqube due to squid rule S2162
In short using instanceof to in equals to check class triggers a warning because subclasses of ClassName would pass the instanceof conditional.
To make the equals method compliant it should null check, then compare getClass of both objects:
if (other == null) {
    return false;
}
....
if (this.getClass() != other.getClass()) {
    return false;
}

I think the implications of both approaches were well understood by the person that submitted this code originally, and I think they were inspired by Effective Java.
Here's a good summary of Josh Bloch's thoughts on the subject:
https://www.artima.com/intv/bloch17.html
This is a tricky one. Given the arguments on both sides, I prefer getClass, however there's a backwards compatibility concern to throw into the mix now, which makes me reticent to merge your PR.

Thanks for the link; the debate between getClass and instanceof had completely passed me by until now!
Having read Bloch's, and others, arguments for it I'm still in the getClass camp; instanceof breaking the symmetry of equals is a killer for me
I'd assumed it was an oversight rather than a design decision; if you have concerns about backwards compatibility bounce the request; it obviously would change behaviour for people relying on superclass equaling subclasses

I am curious if this is worth a flag.

authors: beresfordt ctrimble joelittlejohn


text:
Think about to fix the issues found by adapter checker

I am an automatic service that looks for possible errors in ioBroker and creates an issue for it. The link below leads directly to the test:
https://adapter-check.iobroker.in/?q=https://raw.githubusercontent.com/joergzdarsky/ioBroker.syncthing

[ ] [E200] Not found on npm. Please publish

I have also found warnings that may be fixed if possible.

[ ] [W113] Adapter should support compact mode
[ ] [W400] Cannot find "syncthing" in latest repository

Thanks,
your automatic adapter checker.
P.S.: There is a community in Github, which supports the maintenance and further development of adapters. There you will find many experienced developers who are always ready to assist anyone. New developers are always welcome there. For more informations visit: https://github.com/iobroker-community-adapters/info

Thanks, that all bugs have been fixed.


text:
Compalitible with Windows? Steps for installation in Windows

Is Joern compatible with Windows 10? If yes, what are the steps to be followed? I couldnt find the steps for Windows in the documentation or not sure if I am missing something.

All frontends and the joern cli are compatible, yes.
Just git clone the repo and run sbt stage there. You will find executables, e.g.,:

for joern cli: path\to\joern\joern-cli\target\universal\stage\bin\ammonite-bridge
for c2cpg: path\to\joern\joern-cli\frontends\c2cpg\target\universal\stage\bin

Thank you for the steps. I tried this steps but getting this error (shown below).

$ sbt stage
Picked up _JAVA_OPTIONS: -Xmx512M
copying runtime jar...
mkdir: cannot create directory ‘’: No such file or directory
Picked up _JAVA_OPTIONS: -Xmx512M
Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size
Picked up _JAVA_OPTIONS: -Xmx512M
Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size

I am not sure if this is because of the available amount of memory.
If I have to configure JVM to avoid this error, what would be the recommended memory size?

looks like you should clear or redefine some of your environment variables...

Try to set -Xmx to something larger, 4G or 8G should do it depending on your system.

repo: joernio/joern


text:
:hammer: Add devcontainer

Provide a develop enviroment docker container.
Developers can use it and don't need to deploy the build environment again.

What @ursachec said. Thank you for sharing, but we have a few questions.
It would be good to get a better description of the use case, ideally in the readme, e.g. in a development section.
The base image is almost 1g, comes from a vendor and ships their jdk:
[root@cf01afb1383b TencentKona-11.0.15.b1]# bin/java -version
openjdk version "11.0.15-ga" 2022-04-28
OpenJDK Runtime Environment TencentKonaJDK (build 11.0.15-ga+1)
OpenJDK 64-Bit Server VM TencentKonaJDK (build 11.0.15-ga+1, mixed mode)

I don't know much about them, but can't we just use a more common/standard and/or small image like openjdk or some alpine linux?

@ursachec
It is easy to deploy the build environment with devcontainer and VSCode extension remote - container. And developers don't need to care about the build environment again and can immediately focus on their work.
Compared to such benefits, the cost of the choice of the base image and the JDK distribution is relatively small.

What @ursachec said. Thank you for sharing, but we have a few questions. It would be good to get a better description of the use case, ideally in the readme, e.g. in a development section.
The base image is almost 1g, comes from a vendor and ships their jdk:
[root@cf01afb1383b TencentKona-11.0.15.b1]# bin/java -version
openjdk version "11.0.15-ga" 2022-04-28
OpenJDK Runtime Environment TencentKonaJDK (build 11.0.15-ga+1)
OpenJDK 64-Bit Server VM TencentKonaJDK (build 11.0.15-ga+1, mixed mode)

I don't know much about them, but can't we just use a more common/standard and/or small image like openjdk or some alpine linux?

TencentOS and TencentKona are just the software I use more often.
If you have a better recommendation, I can try to modify it here.

yes, centos and/or openjdk sound good

Ok, I update the devcontainer with them now.

Thank you! Now would you mind adding a few words on how to use it, e.g. in the readme? That'll also give it visibility to others.

Thank you! Now would you mind adding a few words on how to use it, e.g. in the readme? That'll also give it visibility to others.

DONE

Thank you! I'll merge this as is and will add a section on Intellij, which is the IDE most of our team use. Good to get an overview and instructions for different dev setups.

done


text:
feat: typescript setup and working example

closes #10

:tada: This PR is included in version 0.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
[REQUEST] mod type explanation

Hoy there ! Is it possible to link / make a quick tutorial about mod type and item mod ?
the few lines on GBA thread lacks of everything, nothing is explained really

The stuff on GBA is all i know myself tbh, I just exposed the values in memory. GBA is the only source of what they actually do i'm afraid.

authors: XenocodeRCE joffnerd

repo: joffnerd/botw-trainer


text:
Try to generate moq in module mode again

https://github.com/matryer/moq/issues/84 was recently closed, we should update the version and try to generate without the GOPATH again.

Closed in #75

repo: johanbrandhorst/certify


text:
S3 Select mocking support

Hello,
I am wondering if it's possible to deal with a code which uses "S3 Select" feature.
S3 Select is an S3 feature to get contents with SQL query. It is, in general, a tough challenge with a great benefit to test a query like SQL which leads its query complex, and S3 Select is something special in testing difficulty. S3 Select is used with large aws-sdk's interface and its backend is locally unavailable.
My questions:

Is it in gofakes3's scope of concern to support select feature?
If so, how the implementation supposed to be?

I am not sure even it is good idea to use other project's SQL parser for this purpose.

Is it in gofakes3's scope of concern to support select feature?

Yep I think so, but I think it will be difficult to get right. Compliance will be especially tricky - even just the list of different errors that the S3 SQL parser can emit is very long, and that's before we start getting into all the corner cases of the expression parsing. It'll take a long time to replicate the behaviour exactly. Doesn't mean we shouldn't try, but we should be very up front with users that GoFakeS3's implementation will not be able to be accurate for a long time, if ever.

If so, how the implementation supposed to be?

I've got a branch here that shows how I think it should be structured, though this code is in no way ready for release yet and I haven't actually got the SQL part of it up and firing yet, just the structure: https://github.com/johannesboyne/gofakes3/compare/master...shabbyrobe:feature/s3-select?expand=1
It has very minimal impact on the existing code though. S3 Select is quite broad so I think we should focus on shipping a version with support for a small set of use cases and encouraging people to make their own more tailored plugin if they need more.

Same with github.com/araddon/qlbridge. I suppose I shouldn't be surprised; S3 Select is a bit outside the standard. I think we'll probably have to either hand-roll a parser or borrow the bulk of one from an existing project; I'd prefer the latter if possible. I'll take a look around inside qlbridge to see if the guts of it can be airlifted in.

Nice to see your implementation! That's big.

It'll take a long time to replicate the behaviour exactly
Yes, it must be. For testing purpose, it is not always needed to be exact same error message, though. I believe that handling the working example of  let's say, select, will come on the most needed part at first, especially seeing whether where part logic works as intended, for example.

Parsing S3 Select query is quite difficult because it is outside the standard as you pointed out but minio has already implemented [https://github.com/minio/minio/blob/master/pkg/s3select/sql/parser.go]. The approach of parser implementation is quite different with your PEG's one, yet I wonder if it is possible to borrow from them.

Oh that's a good find, I'll see if I can airlift minio's implementation directly. The PEG was just a "let's get started quickly" experiment. I've got the absolute basics almost working locally, but if minio's works nicely out of the box with the structure I added the other day, it makes no sense to continue with my own.

authors: iori-yja shabbyrobe


text:
2.5.0 unreleased

Any chance you could push a release? Thanks much!

sorry, missed this.. pushed to npm a while ago

Thank you!

authors: john-doherty xian

repo: john-doherty/long-press-event


text:
chore(main): release 3.0.1

:robot: I have created a release beep boop
3.0.1 (2023-07-10)
What's Changed

ci: comment on pr with stackblitz link by @johnhwhite in https://github.com/johnhwhite/ng-keyboard-sort/pull/74
build: remove nx dependency by @johnhwhite in https://github.com/johnhwhite/ng-keyboard-sort/pull/73

Full Changelog: https://github.com/johnhwhite/ng-keyboard-sort/compare/v3.0.0...v3.0.1

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/johnhwhite/ng-keyboard-sort/releases/tag/v3.0.1 :sunflower:

repo: johnhwhite/ng-keyboard-sort


text:
Ability to Enable/Disable for Projects using .vscode/settings.json

Volar sometimes interfere with other projects that it should not. I want to disable/enable Volar based on project using .vscode config file.
I saw Deno also uses the technique to Enable/Disable Deno server.
// .vscode/settings.json

{
  "deno.enable": true,
  "deno.unstable": true
}

For Volar, I think we should do something like this too.
// .vscode/settings.json

{
  "volar.enable": true,
}

If it's already implemented, please let me know how to achieve that.

Please track #https://github.com/microsoft/vscode/issues/40239

authors: johnsoncodehk seanghay

repo: johnsoncodehk/volar


text:
Feature/solve catalan translations

Check all catalan country names

Thank you very much for the PR!

repo: joielechong/CountryCodePicker


text:
Create Header component

Must handle the following:

[ ] Name of the chat (.header-title)
[ ] Number of users (.header-users-count)
[ ] Number of messages (.header-messages-count)
[ ] The date of the last message (.header-last-message-date in the format dd.mm.yyyy hh:mm:ss)

To count the number of users in the chat I can create a set that can include only unique values. While iterating through the collection I will append the value of the userId field and after the iteration, the loop will stop the len of the collection will be the number of the users in the chat.
`var arrayExample = [
{
createdAt: "2020-07-16T19:48:12.936Z",
editedAt: "",
id: "80f08600-1b8f-11e8-9629-c7eca82aa7bd",
text: "I don’t *** understand. It's the Panama accounts",
user: "Ruth",
userId: "9e243930-83c9-11e9-8e0c-8f1a686f4ce4"
},
{
createdAt: "2020-07-16T19:48:42.481Z",
editedAt: "",
id: "80f08600-1b8f-11e8-9629-c7eca82aa7bd",
text: "I don’t *** understand. It's the Panama accounts",
user: "Ruth",
userId: "9e243930-83c9-11e9-8e0c-8f1a686f4ce4"
},
];
var usersNumber = new Set();
arrayExample.forEach(message => {
usersNumber.add(message.userId);
});
console.log(usersNumber.length);
Unfortunately _.length_ can only be used for the array or the string, that's why the array will be used instead of setvar usersNumber = new Array();
arrayExample.forEach(message => {
if (!usersNumber.includes(message.userId)){
usersNumber.push(message.userId);
}
});
console.log(usersNumber.length);`


text:
Unable to read mbeans from remote node

Jolokia version :1.3.7
I configured via Jolokia agent servlet in our application which is deployed in Node A of Jetty server  and trying to read jmx beans in another remote Node B using proxy request as follows-
Web.xml
     <servlet>
      <servlet-name>jolokia-agent</servlet-name>
      <servlet-class>org.jolokia.http.AgentServlet</servlet-class>
      <load-on-startup>1</load-on-startup>
      <init-param>
            <param-name>debug</param-name>
            <param-value>true</param-value>
        </init-param>
    </servlet>

    <servlet-mapping>
      <servlet-name>jolokia-agent</servlet-name>
      <url-pattern>/jolokia/*</url-pattern>
    </servlet-mapping>

Javacript
var data = {
                "type": "read",
                "mbean": "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec",
                "attribute" : "Count",
                "target": { "url":"service:jmx:rmi:///jndi/rmi://xx.xxx.xxx.xx:xxxx/jmxrmi" }
            };
var jolokia = new Jolokia({url: "./jolokia", method: "post"});
jolokia.request(data);

If i set target url to invalid hostname, request never fails.
Using JConsole I am able connect using above host name & port
What am I missing here?  Thanks

Could you try first to use a direct request to your JMX proxy (e.g. via postman from within the browser or via curl) ? Does this work ? Could you also try a well known MBean like in the example of https://jolokia.org/reference/html/protocol.html#protocol-proxy ?

curl -X GET http://10.252.136.101:1080/oehcsui/jolokia/read/java.lang:type=Memory/HeapMemoryUsage/  -H 'authorization: Basic YWRtaW46V2VsY29tZS0x'
{"request":{"mbean":"java.lang:type=Memory","attribute":"HeapMemoryUsage","type":"read"},"value":{"init":247463936,"committed":240648192,"max":3504865280,"used":61753920},"timestamp":1505908386,"status":200}
curl -X POST http://10.252.136.101:1080/oehcsui/jolokia/ -H 'authorization: Basic YWRtaW46V2VsY29tZS0x' -d '{ "type": "READ", "mbean" : "java.lang:type=Memory", "attribute" : "HeapMemoryUsage", "target": { "url":"service:jmx:rmi:///jndi/rmi://10.252.137.86:9098/jmxrmi" } }'
{"request":{"mbean":"java.lang:type=Memory","attribute":"HeapMemoryUsage","type":"read","target":{"url":"service:jmx:rmi:///jndi/rmi://10.252.137.86:9098/jmxrmi"}},"value":{"init":247463936,"committed":240648192,"max":3504865280,"used":63001864},"timestamp":1505908548,"status":200}
Exception Curl
curl -X POST http://10.252.136.101:1080/oehcsui/jolokia/ -H 'authorization: Basic YWRtaW46V2VsY29tZS0x' -d '{"type": "READ","mbean": "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec", "attribute" : "Count", "target": { "url":"service:jmx:rmi:///jndi/rmi://10.252.137.86:9098/jmxrmi" } }'
{"request":{"mbean":"kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics","attribute":"Count","type":"read","target":{"url":"service:jmx:rmi:///jndi/rmi://10.252.137.86:9098/jmxrmi"}},"stacktrace":"javax.management.InstanceNotFoundException: kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)\n\tat com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)\n\tat org.jolokia.handler.ReadHandler.doHandleRequest(ReadHandler.java:106)\n\tat org.jolokia.handler.ReadHandler.doHandleRequest(ReadHandler.java:37)\n\tat org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89)\n\tat org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109)\n\tat org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:161)\n\tat org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:99)\n\tat org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:413)\n\tat org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158)\n\tat org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197)\n\tat org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:137)\n\tat org.jolokia.http.AgentServlet$3.handleRequest(AgentServlet.java:420)\n\tat org.jolokia.http.AgentServlet.handleSecurely(AgentServlet.java:301)\n\tat org.jolokia.http.AgentServlet.handle(AgentServlet.java:277)\n\tat org.jolokia.http.AgentServlet.doPost(AgentServlet.java:244)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:707)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:838)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)\n\tat org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)\n\tat org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)\n\tat org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)\n\tat org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621)\n\tat com.oracle.oehcs.web.DefaultFilter.doFilter(DefaultFilter.java:126)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1613)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:541)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1584)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1228)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1130)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:564)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:318)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:112)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)\n\tat org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122)\n\tat org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)\n\tat java.lang.Thread.run(Thread.java:748)\n","error_type":"javax.management.InstanceNotFoundException","error":"javax.management.InstanceNotFoundException : kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec","status":404}

I debugged and noticed that new MBeanServerConnection is not created for target url parameter.

I am able connect & query jconsole via remote process and remote program using JMXServiceURL
List Operation
curl -X POST http://10.252.136.101:1080/oehcsui/jolokia/?maxDepth=2 -H 'authorization: Basic YWRtaW46V2VsY29tZS0x' -d '{ "type": "LIST", "target": { "url":"service:jmx:rmi:///jndi/rmi://10.252.137.86:9098/jmxrmi" } }'
{"request":{"type":"list","target":{"url":"service:jmx:rmi:\/\/\/jndi\/rmi:\/\/10.252.137.86:9098\/jmxrmi"}},"value":{"JMImplementation":{"type=MBeanServerDelegate":"1"},"java.util.logging":{"type=Logging":"1"},"java.lang":{"name=PS Scavenge,type=GarbageCollector":"1","type=Threading":"1","name=PS Old Gen,type=MemoryPool":"1","type=Memory":"1","name=Metaspace,type=MemoryPool":"1","name=PS Survivor Space,type=MemoryPool":"1","type=OperatingSystem":"1","name=PS Eden Space,type=MemoryPool":"1","name=CodeCacheManager,type=MemoryManager":"1","name=Code Cache,type=MemoryPool":"1","name=Compressed Class Space,type=MemoryPool":"1","type=ClassLoading":"1","type=Compilation":"1","name=PS MarkSweep,type=GarbageCollector":"1","name=Metaspace Manager,type=MemoryManager":"1","type=Runtime":"1"},"com.sun.management":{"type=HotSpotDiagnostic":"1","type=DiagnosticCommand":"1"},"jmx4perl":{"type=Config":"1"},"java.nio":{"name=direct,type=BufferPool":"1","name=mapped,type=BufferPool":"1"},"jolokia":{"type=Discovery":"1","type=ServerHandler":"1","type=Config":"1"}},"timestamp":1506018859,"status":200}

@pmandala simply kafka MBeans are not available...
Please reopen if you want to provide more details.

authors: grgrzybek pmandala rhuss

repo: jolokia/jolokia


text:
Game crashes when launching Minecraft with 1.18.2 Forge 3.1.9

Hello MTR Mod,
When using Forge 1.18.2 version of 3.1.9, the game crashes every time when launching with Forge.
Minecraft Version:
ForgeOptifine 1.18.2
MTR Mod Version:
Forge 1.18.2 - mod version 3.19
The Mod list is in the crash reports
CRASH REPORTS:
crash-2022-11-18_21.45.48-client.txt
crash-2022-11-24_11.46.06-client.txt
Please fix this issue. Thank you!

Please update all mods, especially the Joban Client Mod.

https://github.com/jonafanho/Minecraft-Transit-Railway/issues/559

repo: jonafanho/Minecraft-Transit-Railway