text:
Refactor music box configuration file options

          There was a comment/question from Rebecca Buchholz at the Demo yesterday; she was sitting next to me, and you may not have heard it. She asked about using initial_conditions.csv to set the initial species concentrations instead of my_config.json.

Initialization of concentrations involves a large number of floating-point numbers, and I believe that specification would be technically better in tabular form (CSV) instead of JSON. We have a JSON mechanism to include CSV file, rather than inserting many values into the JSON configuration. The CSV file would be better for our users to manage and review.
I know that we are retaining compatibility with the Fortran MusicBox. I recommend that we move forward soon with changes that will increase CSV usage through our product suite.
Originally posted by @carl-drews in https://github.com/NCAR/music-box/pull/214#pullrequestreview-2268955859
Acceptance criteria

All existing configuration files in our examples use the new format
The new format is read correctly
Species configurations are only read from the initial conditions section

the initial conditions section can either be a path to a file
or the initial conditions section can be an in-place csv file, specified as a set of lists of lists. This is useful for music box interactive since we wouldn't need to write out csv files

Evolving conditions can be read from a file

or read directly from the csv, like the initial condition. Again, useful for music box interactive

Ideas

Consider supporting configuration files of these formats

{
    "box model options": {
        "grid": "box",
        "chemistry time step [sec]": 1.0,
        "output time step [sec]": 1.0,
        "simulation length [sec]": 3600.0
    },
    "evolving conditions": {
        "filepath": "evolving_conditions.csv"
    },
    "initial conditions": {
        "filepath": "initial_conditions.csv"
    },
   "mechanism confiuration" : {
       "relative file path": "camp_data/config.json"
   }
}

or, inplace conditions
{
    "box model options": {
        "grid": "box",
        "chemistry time step [sec]": 1.0,
        "output time step [sec]": 1.0,
        "simulation length [sec]": 3600.0
    },
    "evolving conditions": {
        data: [
                      ["ENV.pressure [Pa]", "ENV.temperature [K]", "PHOT.O2_1 [s-1]"],
                      [1000, 200, 0.5],
                  ]
    },
    "initial conditions": {
        data: [
                      ["ENV.temperature", "ENV.pressure", "CONC.H2O", "CONC.CH4"],
                      [1000, 200, 0.5, 1e-4],
                  ]
    },
   "mechanism confiuration" : {
       "relative file path": "camp_data/config.json"
   }
}

Actually, we might consider storing all of the mechanism information in the file as well. That would remove the need to write any files out. We would then have to update the musica API to take json objects, but that is already a goal we have anyways so that we push all of the parsing down to musica. @mattldawson would this be a good time to do that or do you think it's too soon?

Actually, we might consider storing all of the mechanism information in this file as well. That would remove the need to write any files out. We would then have to update the musica API to take json objects, but that is already a goal we have anyways so that we push all of the parsing down to musica. @mattldawson would this be a good time to do that or do you think it's too soon?

I like that idea. Do we need to make any changes to the open-atmos format for this to work?

@mattldawson I don't belive so. open-atmos already assume everything is in the same file. All we have to do is strip out whatever information is in the mechanism and pass it along to musica

Doing just CSV vs. JSON in this ticket; leave choice of solver (Rosenbrock and others) to another issue.

This is also related to #260, but only indirectly

The code base has changed substantially since October. I will re-synch my obsolete dev branch with the latest main.

Here is the in-progress branch for this issue:
https://github.com/NCAR/music-box/tree/220-config-file-csv

My initial conditions for dev/test look like this in the JSON:
"initial conditions": {
"filepath": "initial_conditions.csv",
"data": [
["ENV.temperature [K]", "ENV.pressure [Pa]", "CONC.A [mol m-3]", "CONC.B [mol m-3]"],
[200, 70000, 0.67, 2.3e-9]
]
},

Since "Species configurations are only read from the initial conditions section", then it follows that the chemical species section of my_config.json will be removed and no longer supported. Delete:
  "chemical species": {
    "A": {
      "initial value [mol m-3]": 1
    },
    "B": {
      "initial value [mol m-3]": 0
    },
    "C": {
      "initial value [mol m-3]": 0
    }
  },

@mattldawson and @K20shores :
For the carbon_bond_5 example we have these initial conditions:
    "initial conditions": {
        "initial_reaction_rates.csv": {}
    },

Those are reaction rates, not initial species concentrations. That CSV files looks like this:
EMIS.NO,EMIS.NO2_emis,EMIS.CO,EMIS.SO2,EMIS.FORM,EMIS.MEOH,EMIS.ALD2_emis,EMIS.PAR,EMIS.ETH,EMIS.OLE,EMIS.IOLE,EMIS.TOL,EMIS.XYL,EMIS.ISOP,PHOTO.NO2,PHOTO.O3->O1D,PHOTO.O3->O3P,PHOTO.NO3->NO2,PHOTO.NO3->NO,PHOTO.HONO,PHOTO.H2O2,PHOTO.PNA,PHOTO.HNO3,PHOTO.NTR,PHOTO.ROOH,PHOTO.MEPX,PHOTO.FORM->HO2,PHOTO.FORM->CO,PHOTO.ALD2,PHOTO.PACD,PHOTO.ALDX,PHOTO.OPEN,PHOTO.MGLY,PHOTO.ISPD
1.44e-10,7.56e-12,1.96e-09,1.06e-09,1.02e-11,5.92e-13,4.25e-12,4.27e-10,4.62e-11,1.49e-11,1.49e-11,1.53e-11,1.4e-11,6.03e-12,0.00477,2.26e-06,0.000253,0.117,0.0144,0.000918,2.59e-06,1.89e-06,8.61e-08,4.77e-07,1.81e-06,1.81e-06,7.93e-06,2.2e-05,2.2e-06,1.81e-06,2.2e-06,0.000645,7.64e-05,1.98e-09

Do you intend for us to read the same file twice, one time looking for prefixes like ENV and CONC, and then read it a second time for prefixes like EMIS and PHOTO? We could do that, but I'm thinking it would be easier to support two JSON keywords instead, like:
"initial concentrations":
"initial rates":

The flow tube example also uses "initial conditions" for the reaction rates:
"initial conditions": {
"filepath": "initial_reaction_rates.csv"
},
The reaction rates look like this:
modeling2:/home/drews/MusicBox/music-box-220-config-file-csv/src/acom_music_box/examples/configs/flow_tube> cat initial_reaction_rates.csv
LOSS.SOA1 wall loss.s-1,LOSS.SOA2 wall loss.s-1
0.01,0.05

I would lean toward allowing any initial conditions (rates, concentrations, temperature, pressure, etc.) to be in one file, or multiple files if a user prefers. Something like:
"initial conditions": {
  "filepaths": [ "all_conditions.csv" ]
}

or
"initial conditions": {
  "filepaths": [
    "some_concentrations.csv",
    "other_concentrations.csv",
    "rates_and_environmental.csv"
  ]
}

What do you think?

Thanks @mattldawson. The python code will not know any significance to the filenames, and we could have filepaths like:
concentrations_CSU.csv
BoulderReservoir-20241230.csv
fromCESM-Pinatubo05.csv
Mars-STP.csv
I do like the idea of not tossing any and all initial conditions into the same CSV file, and letting the research pick her own filenaming conventions to reflect her current topic of research. So:

The Python code will read all the filepaths specified in initial conditions.
The Python code may take multiple passes through the set of files, and collect the relevant information from each one.
Python will collect the relevant information by header prefix. Probably ENV and CONC will refer to concentrations, and EMIS and PHOTO will refer to reaction rates.

Multiple passes would make the code more modular but slower.

authors: K20shores carl-drews mattldawson

repo: NCAR/music-box


text:
Detect EMAIL_ADDRESS

Description:
An email address is a type of PII that can directly reveal personal details such as identity, contact information, or even location of the individual.
Detection Steps:
An Email Address basically follows the standard format of having a username (which could include letters, numbers, and special characters). Then the @ symbol which separates the username and domain. Lastly would be the domain, which also contains letters, numbers and special characters followed by .com, .org, or etc.

yes, that is why i put etc after listing some examples, like the list goes on but i did not want to put all of them if that make sense.

This is a little more complicated as there is an RFC that defines valid email addresses, including characters and length.  Wikipedia has a good write-up on it

source: gharchive/issue

license: MIT

license_type: permissive

license_source: github-api


text:
Hotfix/api search

Description:

Fix API functions misusing and field typo.

Changes:

Fix ArticleAPI.search function call (missing params)
Change name 'timeQuery' to 'tbs' in API query.

Build Status
✅

LGTM

source: gharchive/pull-request

repo: NCTU-SYNC/sync-frontend


text:
[ToDo] インストーラにデジタル署名をつけるには？（優先度低）

タイトルの通り．
っていうか NCVC.exe にもいる？

Issueを上げるリポジトリを間違えたｗｗｗ


text:
[SD ENHANCEMENT] Rework AA for FCO_CPC_EXT.1 to follow standard TSS/Guidance groupings

Provide the location of the issue
Section 3.5.1 para 260 (Assurance activities for FCO_CPC_EXT.1.)
What is the enhancement request for the cPP? Please describe.
Please clarify if the evaluator is expected to report findings/results of investigating issues a) through i). Please also consider reworking these activities to follow standard TSS/Guidance categories.
Describe the solution you'd like
As stated in the comment.
Additional context
This command and solution were submitted by the Canadian Centre for Cyber Security. [Reference CCCS-SD-35]

Note:  The first part of the request in the issue description should be a non-issue since paragraph 260 already ends with, “… and shall report the answers.”

repo: ND-iTC/Documents


text:
Includes apex package from CRAN

This is starting to be used in some popgenInfo vignettes.
Fixes #7.

Passes Docker hub build.

repo: NESCent/popgen-docker

license: mit

license_source: bigquery


text:
Pre-processing issues

Loaded
https://s3.uni-jena.de/nmrxiv/local/archive/f5077a50-513a-4595-9f42-92c5726e62f3/l-Linalool.zip
Initial:
nmrium-initial.json
After
nmrium.json
https://github.com/NFDI4Chem/nmrium-react-wrapper/assets/4065285/8240b5a7-2b96-4077-a4b1-3a057af3d50a

@CS76
It appears that there is an issue when remapping the nmrium object. It's wrong that x, im, and re are present in the info object instead of the expected data object. Additionally, you are still saving the modified information (info object) instead of the original information (originalInfo object)

Dear all,
for me this seems to be connected to NMRium requiring the processed spectra files in EXPNO/PDATA/PROCNO folder of Bruker NMR data (expect for some types which get automated processing in NMRium) and is connected to possibly also to this issue https://github.com/cheminfo/nmrium/issues/2830 .
This processed spectra are not always present, because older version of TopSpin (somewhere between TopSpin 3.6 and 3.2) do not necessarily safe this type of data in a Bruker folder.  Presumably the data providede above is from NMRShiftDB and they also have older data not having the processed spectra in PRCNO folders.
Best,
Tillmann

@hamed-musallam - here is another case

authors: CS76 hamed-musallam tilfischer


text:
nz-step的组件当(nzIndexChange)事件存在时[nzDisabled]失效

Reproduction link
https://ng-zorro-antd-start-anenfg.stackblitz.io/
Steps to reproduce
(nzIndexChange)存在的时候，[nzDisabled]="false"失效
What is expected?
onIndexChange(index: number): void {
this.index = index;
}事件存在的时候，[nzDisabled]="false"的效果是不可以点击的
What is actually happening?
onIndexChange(index: number): void {
this.index = index;
}事件存在的时候，[nzDisabled]="false"的效果是可以点击的

Environment
Info

ng-zorro-antd
8.5.2

Browser
谷歌

@007GitHu [nzDisabled]="false" 的意思本来就是可点击

repo: NG-ZORRO/ng-zorro-antd


text:
feat(module:input-number): support formatter & parser

PR Checklist
Please check if your PR fulfills the following requirements:

[x] The commit message follows our guidelines: https://github.com/NG-ZORRO/ng-zorro-antd/blob/master/CONTRIBUTING.md#commit
[ ] Tests for the changes have been added (for bug fixes / features)
[x] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Application (the showcase website) / infrastructure changes
[ ] Other... Please describe:

What is the current behavior?

Issue Number: N/A
What is the new behavior?
Does this PR introduce a breaking change?
[ ] Yes
[x] No

Other information

Coverage decreased (-0.03%) to 40.828% when pulling 908ffd782ad2b5304ca5c59a5675dc84c2540a2d on vthinkxie:support-input-number-formater into 504fdc8c57436b46e2914105c7f394025e783260 on NG-ZORRO:master.

authors: coveralls vthinkxie


text:
Det skal ikke være så mye distanse mellom avsnittene

Sammenlign nifutypst-malen:

Med publisert rapport:

Fikset!

repo: NIFU-NO/nifutypst


text:
known issue: high severity vulnerabilities tracing to jpeg-js

first of all: thanks a lot for this plugin!
when installing it via npm, it gives a warning for 6 high severity vulnerabilities.
i'll probably use it anyway, but maybe it would be a good idea to fix them at some stage :)

Thanks for the note!  When I wrote this, there were no vulnerabilities, but it looks like jpeg-js (from png-to-ico) now has an infinite loop vulnerability.
This should not be unsafe for running this plugin, as the plugin is only active at build time, so it wouldn't cause a DoS on a website.
Regardless, I will look into fixing this shortly.  Thanks again for raising it.
$ npm install --save-dev eleventy-plugin-gen-favicons                                                                                [19:07:01]

added 118 packages, and audited 119 packages in 15s

9 packages are looking for funding
  run `npm fund` for details

6 high severity vulnerabilities

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.

$ npm audit                                                                                [19:07:28]
# npm audit report

jpeg-js  <0.4.4
Severity: high
Infinite loop in jpeg-js - https://github.com/advisories/GHSA-xvf7-4v9q-58w6
No fix available
node_modules/jpeg-js
  @jimp/jpeg  <=0.12.0 || >=0.16.1
  Depends on vulnerable versions of jpeg-js
  node_modules/@jimp/jpeg
    @jimp/types  <=0.11.1-canary.891.908.0 || >=0.16.1
    Depends on vulnerable versions of @jimp/jpeg
    node_modules/@jimp/types
      jimp  0.3.6-alpha.5 - 0.11.1-canary.891.908.0 || >=0.16.1
      Depends on vulnerable versions of @jimp/types
      node_modules/jimp
        png-to-ico  >=2.0.1
        Depends on vulnerable versions of jimp
        node_modules/png-to-ico
          eleventy-plugin-gen-favicons  *
          Depends on vulnerable versions of png-to-ico
          node_modules/eleventy-plugin-gen-favicons

6 high severity vulnerabilities

Some issues need review, and may require choosing
a different dependency.

There is a related issue in the png-to-ico project: https://github.com/steambap/png-to-ico/issues/19
and the related issue in jimp: https://github.com/oliver-moran/jimp/issues/1088
It sounds like the png-to-ico library has not been updated yet, so I can't simply bump the version.
However, as the author confirms there, only the png part of the library is used, so hopefully the library is not susceptible to this issue.
I will investigate whether I can manually force a different version of jpeg-js, or maybe at least swap out jimp-compact as described in the issue above.  Ideally, this would be fixed by the jimp project updating to a newer version of jpeg-js.  Not sure what the timeline is on that :( .

png-to-ico has been updated.  I will release a new version picking that up this week.
https://github.com/steambap/png-to-ico/pull/21

I have released version 1.1.1 which now shows no vulnerabilities (commit c4c9572f9d7d46427d631682faac5c7d251bcdf9 ).  Thanks again!

authors: NJAldwin changethe

repo: NJAldwin/eleventy-plugin-gen-favicons


text:
Added TimeSpan handling

..

https://github.com/NJsonSchema/NJsonSchema/commit/57839b8a74a61002b3c2116918556e87c4bb4dae

repo: NJsonSchema/NJsonSchema


text:
Ensure data completeness

Users of Routinator should be able to get confirmation through the monitoring endpoint that all data was retrieved and processed correctly.

As you can see, in the following example, the second session was 'stale' and didn't updated the (juniper) router with the latest state of the DB.
Inconsistencies can lead to incorrect rejection of (now) valid ROA's.
show validation session            Session                                  State   Flaps     Uptime #IPv4/IPv6 records 178.249.a.b                           Up         12          1w6d 20:09:07 71872/13121 178.249.c.d                           Up          7           1w6d 20:16:23 69882/12806

Hi @ebais,
Forgive me if the following is incorrect, I have limited experience with the Routinator and with Juniper routers. Does this actually show a problem with the Routinator (presumably the RPKI source being connected to by both sessions) or could it just be showing a connectivity problem between the router and the Routinator instance?
I work for NLnet Labs and as part of my work on the NLnet Labs Gantry project (which aims to test Routinator with virtual service routers) I did some limited testing with the Juniper VMX 18.2R1.9 VSR and the Routinator. In that testing I used the command show validation session detail to retrieve the serial number of the update (e.g. see this example).
Does your Juniper router support the command show validation session detail? If so, what does it show? I'm wondering if, like the Juniper VMX example here, that you can see the serial number of the session (though I'm not clear if that serial number is something captured from the Routinator or just an incremented counter on the router)
I'd be interested to know exactly which Juniper model you are using. We could then consider whether it makes sense to test against that specific version of if we should update the Gantry test to run with two connected Routinators for a longer time period to replicate this kind of session variance.
Thanks,
Ximon

There was a difference in the serial in this particular case, yes.
The correct session had a serials of 254, while the stale session was stuck at 34 :

show validation session detail
Session 178.249.a.b, State: up, Session index: 2
Group: rpki-validator, Preference: 100
Local IPv4 address: 5.10.a.b, Port: 3323
Refresh time: 120s
Hold time: 240s
Record Life time: 3600s
Serial (Full Update): 254
Serial (Incremental Update): 254
Session flaps: 5
Session uptime: 1w6d 20:08:41
Last PDU received: 00:00:38
IPv4 prefix count: 71872
IPv6 prefix count: 13121
Session 178.249.c.d, State: up, Session index: 3
Group: rpki-validator, Preference: 100
Local IPv4 address: 5.10.c.d, Port: 3323
Refresh time: 120s
Hold time: 240s
Record Life time: 3600s
Serial (Full Update): 34
Serial (Incremental Update): 34
Session flaps: 5
Session uptime: 1w6d 20:15:58
Last PDU received: 00:01:01
IPv4 prefix count: 69882
IPv6 prefix count: 12806

Thanks @ebais!

#122 adds three new gauges to the monitoring data that show how long ago an update was last started, how long ago an update last finished successfully, and how long the last update took:
# HELP last_update_start seconds since last update started
# TYPE gauge
last_update_start 20

# HELP last_update_duration duration in seconds of last update
# TYPE gauge
last_update_duration 17

# HELP last_update_done seconds since last update finished
# TYPE gauge
last_update_done 2

Using this data, an alert can be set up when the last update was started longer than --refresh seconds ago.

These new gauges work well in Prometheus/Grafana and allow setting up alerts when validation takes overly long. Thanks!

authors: AlexanderBand ebais partim ximon18

repo: NLnetLabs/routinator


text:
[question] asp.net core + nlog fluent api

Hello,
I didn't find it in documentation so asking here.
I have asp.net core api where I configured nlog (using documentation https://github.com/NLog/NLog/wiki/Getting-started-with-ASP.NET-Core-2)
I would like to use fluent api https://github.com/NLog/NLog/wiki/Fluent-API. Is there any option to use fluent api with ILogger<T> interface? I know I can use NLog.LogManager.GetCurrentClassLogger() instead of injected ILogger<T> but maybe there is another option?

The NLog.Fluent-extension methods are for NLog own Logger-object.
You are welcome to create your improved extension methods for Microsoft ILogger-object.
Maybe can be some inspiration: https://github.com/NLog/NLog.Extensions.Logging/wiki/NLog-properties-with-Microsoft-Extension-Logging

@krzSobin Does it answer your question?

authors: krzSobin snakefoot

repo: NLog/NLog


text:
Logger - Added missing MessageTemplateFormatMethodAttribute + removed obsolete internal method

Follow up to #2345 and #2406

ok thx!


text:
Use optimal 2-d array indexing in mc_reach

Several corrections were made in response to the original comment. A separate, simple PR should be used to fix this last holdout (and any others that might have crept in) in mc_reach.pyx:
212             flowveldepth[fill_index][idx] = val
to be this:
 212             flowveldepth[fill_index, idx] = val

Try using buf_view[i, j] instead of buf_view[i][j] for all of these memoryview accesses. If my memory is correct, buf_view[i][j] will create an intermediate object (buf_view[i] evaluates, then tmp_obj[j]).  Maybe cython is smart enough to optimize that away, but I doubt it.

Originally posted by @groutr in https://github.com/NOAA-OWP/t-route/pull/224#discussion_r565835333

New array indexing has long been adopted in mc_reach.

authors: JurgenZach-NOAA jameshalgren

repo: NOAA-OWP/t-route


text:
stray lines from GO LAND for axes longer then modulo length

Reported by andrew wittenberg on 16 Feb 2014 22:52 UTC
Not sure if this is an issue with land.jnl, or in Ferret itself.
 	NOAA/PMEL TMAP
 	FERRET v6.86
 	Linux 2.6.32-431.3.1.el6.x86_64 64-bit - 01/16/14
 	16-Feb-14 17:44

yes? set reg/x=0:540; go basemap; go land

Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/2141

Comment by ansley.b.manke on 18 Feb 2014 19:05 UTC
It must have to do with land.jnl, or with the dataset geo_borders.cdf which "go land" uses.

Comment by ansley.b.manke on 24 Feb 2014 20:56 UTC
Fixed in plotxy.F and plotxy_ribbon.F, handling modulo longitude, when there is a longitude branch point or more than one out in the middle of the plot.

Attachment from andrew.wittenberg on 16 Feb 2014 22:54 UTC
The resulting plot.
REPLACE THIS TEXT WITH UPLOADED FILE ./attachments/TRAC_2141_GIT_1413/plot.gif

Attachment from ansley.b.manke on 18 Feb 2014 19:04 UTC
with "go land_detail"
REPLACE THIS TEXT WITH UPLOADED FILE ./attachments/TRAC_2141_GIT_1413/plot_detail.gif

repo: NOAA-PMEL/Ferret

license: Unlicense


text:
%xaxis and %yaxis do not apply the 5th argument, # small tics

Reported by Ansley B Manke on 12 Dec 2003 20:03 UTC
Draw two axes, changing the requeste number of tics:
yes? ppl %xaxis/nouser 1,100,10,0.2,4
yes? ppl %xaxis/nouser 1,100,10,0.4,6
Does not set the tics.
Note this bug is a typo-bug in the routine ppl/plot/prmtiv.F
the parameter NSMTCS from the command argument, but then calls
the axis-drawing routine using (undefined) NSMTCX.
Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/747

Comment by Ansley.B.Manke@noaa.gov on 15 Dec 2003 17:20 UTC
fixed in routine ppl/plot/prmtiv.F  and checked in on
the bug-fix branch of Ferret v5.53


text:
Add docstrings and make them available in documenter.jl build

docs/src/api/public.md now has a list of public documented methods with a custom sorting
No "mainline" documentation is useful yet (quickstart, examples, etc)
Added documentation to several key PRAS functions (Shortfall, SequentialMonteCarlo, assess, SystemModel). I also re-export these and include them in the public documentation.

Fixes #13


text:
Solar water heating (SWH) system capacity equation unclear

The solar water heating model displays the "Rated system size" on the Solar Water Heating input page. This value is passed to SSC as system_capacity.
The units don't make sense in the description in Help and equation in the UI form equation. See this post on the SAM forum for details:
https://sam.nrel.gov/forum/forum-general/4299
For model references, see https://sam.nrel.gov/solar-water-heating.html (equations based on Duffie and Beckman).

I revised the Help description for Fall 2023:

repo: NREL/SAM


text:
[Feature]: add relevant icons to each skill in resources page

Is your feature request related to a problem? Please describe.

Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response

Please assign me this issue.

hey, i have added the icons , can you please review and merge this.

Hello , I hope you're doing well. I'm interested in contributing to this task. I am eager to help in any way I can and believe I can make valuable contributions to the project. I humbly request you to assign me with the task.

repo: NSCC-BPIT/NSCC-BPIT-Website


text:
ENH: add detector motors

We need to have ophyd objects for the detector motors, this way we can make certain that the detector was in the same place when foreground and background images were taken.

I'm moving this off this cycle's release.


text:
Add A2scan, D2scan, MeshScan

Holding off on these until the algebra is ready

Closed by #139

repo: NSLS-II/bluesky


text:
xspress3 error after power cycle

This morning we had to power cycled Xspress3 and afterwards we're having problem with running it in bluesky.
It shows:
TimeoutError: Failed to connect to XF:05IDD-ES{Xsp:1}:HDF5:NumCapture_CALC.DISA
when we tried to run our regular xanes script.
BlueSky [3]: xanes(erange = [10207-50, 10207-20, 10207+50, 10207+120],
            estep = [2, 1, 5],
            harmonic = None,
            acqtime=0.2, roinum=1, i0scale = 1e8, itscale = 1e8,samplename='W_test',filename='W_test')
Run aborted
Failed to connect to XF:05IDD-ES{Xsp:1}:HDF5:NumCapture_CALC.DISA
---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
<ipython-input-3-71bf3df98a35> in <module>()
      2             estep = [2, 1, 5],
      3             harmonic = None,
----> 4             acqtime=0.2, roinum=1, i0scale = 1e8, itscale = 1e8,samplename='W_test',filename='W_test')

/nfs/xf05id1/ipython_ophyd/profile_xf05id1/startup/92-xanessetup.py in xanes(erange, estep, harmonic, correct_c2_x, correct_c1_r, acqtime, roinum, i0scale, itscale, samplename, filename)
    186
    187     #run the plan
--> 188     scaninfo = gs.RE(xanes_scanplan, livecallbacks, raise_if_interrupted=True)
    189
    190     #close b shutter

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/bluesky/run_engine.py in __call__(self, plan, subs, raise_if_interrupted, **metadata_kw)
    568                 exc = self._task.exception()
    569                 if exc is not None:
--> 570                     raise exc
    571             if raise_if_interrupted and self._interrupted:
    572                 raise RunEngineInterrupted("RunEngine was interrupted.")

/home/xf05id1/.conda/envs/collection/lib/python3.4/asyncio/tasks.py in _step(***failed resolving arguments***)
    237                 # We use the `send` method directly, because coroutines
    238                 # don't have `__iter__` and `__next__` methods.
--> 239                 result = coro.send(None)
    240             else:
    241                 result = coro.throw(exc)

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/bluesky/run_engine.py in _run(self)
    740             logger.error("Run aborted")
    741             logger.error("%s", err)
--> 742             raise err
    743         finally:
    744             self.state = 'idle'

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/bluesky/run_engine.py in _run(self)
    714                         raise
    715                     except Exception as e:
--> 716                         self._genstack[-1].throw(e)
    717                     self.log.debug("Response: %r", response)
    718                 except KeyboardInterrupt:

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/bluesky/plans.py in __iter__(self)
     57         # Some metadata is compute in _pre, so it must be done before
     58         # the RunStart is generated by open_run, below.
---> 59         yield from self._pre()
     60
     61         # If the plan has a 'subs' attribute, yield a 'subscribe' Msg for

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/bluesky/plans.py in _pre(self)
     98                 root = obj.root
     99                 if root not in staged:
--> 100                     yield Msg('stage', root)
    101                     staged.append(root)
    102

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/bluesky/run_engine.py in _run(self)
    710                         coro = self._command_registry[msg.command]
    711                         self.log.debug("Processing %r", msg)
--> 712                         response = yield from coro(msg)
    713                     except KeyboardInterrupt:
    714                         raise

/home/xf05id1/.conda/envs/collection/lib/python3.4/asyncio/coroutines.py in coro(*args, **kw)
    204         @functools.wraps(func)
    205         def coro(*args, **kw):
--> 206             res = func(*args, **kw)
    207             if isinstance(res, futures.Future) or inspect.isgenerator(res):
    208                 res = yield from res

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/bluesky/run_engine.py in _stage(self, msg)
   1206         if not hasattr(obj, 'stage'):
   1207             return
-> 1208         result = obj.stage()
   1209         self._staged.add(obj)  # add first in case of failure below
   1210         return result

/home/xf05id1/Repos/hxntools/hxntools/detectors/xspress3.py in stage(self)
    654         self._abs_trigger_count = 0
    655         self._acquisition_signal.subscribe(self._acquire_changed)
--> 656         super().stage()
    657
    658     def unstage(self):

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/ophyd/device.py in stage(self)
    393                 device = getattr(self, attr)
    394                 if hasattr(device, 'stage'):
--> 395                     device.stage()
    396                     devices_staged.append(device)
    397         except Exception:

/home/xf05id1/Repos/hxntools/hxntools/detectors/xspress3.py in stage(self)
    223
    224         # actually apply the stage_sigs
--> 225         ret = super().stage()
    226
    227         self._fn = self.file_template.get() % (self._fp,

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/ophyd/areadetector/filestore_mixins.py in stage(self)
    160         set_and_wait(self.file_name, filename)
    161         set_and_wait(self.file_number, 0)
--> 162         super().stage()
    163
    164         # AD does this same templating in C, but we can't access it

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/ophyd/areadetector/filestore_mixins.py in stage(self)
     80         self._locked_key_list = False
     81         self._datum_uids.clear()
---> 82         super().stage()
     83
     84

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/ophyd/areadetector/plugins.py in stage(self)
     89         # Ensure the plugin is enabled. We do not disable it on unstage.
     90         set_and_wait(self.enable, 'Enable')
---> 91         super().stage()
     92
     93     @property

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/ophyd/device.py in stage(self)
    372
    373         # Read current values, to be restored by unstage()
--> 374         original_vals = {sig: sig.get() for sig, _ in self.stage_sigs.items()}
    375
    376         # We will add signals and values from original_vals to

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/ophyd/device.py in <dictcomp>(.0)
    372
    373         # Read current values, to be restored by unstage()
--> 374         original_vals = {sig: sig.get() for sig, _ in self.stage_sigs.items()}
    375
    376         # We will add signals and values from original_vals to

/home/xf05id1/.conda/envs/collection/lib/python3.4/site-packages/ophyd/signal.py in get(self, as_string, **kwargs)
    333             if not self._read_pv.wait_for_connection():
    334                 raise TimeoutError('Failed to connect to %s' %
--> 335                                    self._read_pv.pvname)
    336
    337         ret = self._read_pv.get(as_string=as_string, **kwargs)

TimeoutError: Failed to connect to XF:05IDD-ES{Xsp:1}:HDF5:NumCapture_CALC.DISA

Resolved with @imrealkaren and @gjwillms. Their xspress3 IOC will need some work this next long shutdown.

authors: imrealkaren klauer


text:
Jupyter Inline/Notebook "is Loading..." but is actually frozen

It seems that sometimes a very simple Colab ipynb (with only simple statements like a=2) cannot be loaded onto the Jupyter Notebook Activity page.
And it seems once the Colab ipynb is loaded to the Activity page, any update in the change on the Colab's side will be saved and updated in the Activity side as well (which is nice). However, the same ipynb on Colab could be un-loadable to the app one minute before!
So what is causing some ipynb to be loading significantly slower or unable to load to Jupyter Activity page?
And "is loading..." is not true, I have found that if a Colab cannot be loaded within the first 10 seconds, then chances are it is actually frozen and will never be loaded.

could you also give link to broken activity, pls?

@Brainenjii
Here are the two examples I encountered today:
Link1:
Colab: https://colab.research.google.com/drive/1ANtBKB5nDGdHc6HWeXKowox9hEz_h1Hx
(with plot from matplotlib, the ipynb wont load)
Link2:
Colab: https://colab.research.google.com/drive/1FWRnfNaka64reOJOOy4J797swZRbOu4b
(with Image from IPython, also wont load)
I have made a path with these Colab links as Path Activities:
Path: https://achievements-dev.firebaseapp.com/#/paths/-LKySkg663NzWlsliMcf
Hope it helps!

Awesome, thx! Seems like issue with logging events:
Error: Reference.push failed: first argument  contains an invalid key (text/plain) in property 'logged_events.otherActionData.payload.problemJSON.cells.1.outputs.0.data'.  Keys must be non-empty strings and can't contain ".", "#", "$", "/", "[", or "]"

Wow! Nice! Where are these logging information found?
And I have just managed to reproduce the most peculiar case today:
Colab Link: https://colab.research.google.com/drive/15RksT0xa6ppt6XRU7akGQEkhudwjivLV
(simple a = 2 kind statements..., but still cannot load!)
Path-Activity: https://achievements-dev.firebaseapp.com/#/paths/-LKySkg663NzWlsliMcf/activities/-LMfqLNw1fhRQ6BmG4Xo

I confirm that any notebook with ANY output for ANY code block will prevent the notebook from loading. Here is a very simple example. @Brainenjii  can you create a Jupyter activity with this notebook and see it load?
https://colab.research.google.com/drive/1FmxVKooEmOQVokfASfcFG2i48WwbTiRB

it seems the problem with python commentaries (strings started with # char)

@Brainenjii I have made some activities that involve print() and text, like this:
https://achievements-dev.firebaseapp.com/#/paths/-LMfAB4SNR25AyoFLmN5/activities/-LMfAM_9vPA2dUy09IkE

and this activity for example works without the frozen loading issue.
So is the problem with print() text?

ok, it seems the problem with cells that prints some text. Jupyter stores text output as {.."text/plain": "Text"..} and firebase forbid \ char using as node
E.g. if we remove that output from your notebook - it'll open but if you'll try to solve it - firebase function will meet the same issue.

I think related to my above comment on print() and text, for your "open" link example, I am also able to solve it:

Any advice on this?

I added little change to avoid frozen state. But it still fails if you'll give plain text output for a node (try to use "true" as the solution in your activity and you'll get an error in development console). However, it allows keeping the solution in spite of this error. For now, I don't understand how it could be so, I'm checking it

ok, found. Firebase function that handles jupyter solutions keeps the solution from lambda function as stringified JSON. It allows receiving solution at browser page. The stored solution has no output, so it allows to keep it as real JSON in firebase

authors: Brainenjii FeynmanDNA scboesch

repo: NUS-ALSET/achievements


text:
Endre høyden på nve-select

Vi har feil høyde på nve-select i forhold til Shoelace, vi bør avtale med designeren hvilken høyde det skal være.
Designsystemet
small -> height: 48
medium-> height: 48
large-> height: 56
Shoelace
small -> height: 30
medium-> height: 40
large-> height: 50

Vi følger dimensjonskalen som er laget for NVE. Ikke shoelace sin.
Vi går over til 3 størrelser.

Small - 32 px
Medium - 40 px
Large - 48 px

Dette gjelder:

NVE-Input
NVE-select

Flott!
nve-select og nve-input bruker idag disse størrelsene (fra global.css):
--sl-input-height-small: var(--sizing-small);
--sl-input-height-medium: var(--sizing-medium);
--sl-input-height-large: var(--sizing-large);

I stedet for å endre disse, skulle vil heller justert på alle "størrelses-variablene" i designsystemet, slik at også nve-knappen kan få en standard-størrelse som passer med de andre komponentene?
Om vi justerer ned alle størrelses-variablene ett hakk (8px), kan vi droppe sizing-4x-small som uansett ikke brukes av nve-komponentene idag:
--sizing-x-large: 3.5rem; /* 56px */
--sizing-large: 3rem; /* 48px */
--sizing-medium: 2.5rem; /* 40px */
--sizing-small: 2rem; /* 32px */
--sizing-x-small: 1.5rem; /* 24px */
--sizing-2x-small: 1rem; /* 16px */
--sizing-3x-small: 0.5rem; /* 8px */

Vi må kanskje innføre sizing-xx-large hvis vi fortsatt trenger noe som er 64px?
Da kan nve-knappen få størrelser som passer bedre til størrelses-skalaen (small vil bruke sizing-small i stedet for sizing x-small, som den gjør idag, og medium kan være standard.
Det er flere komponenter som vil påvirkes hvis vi endrer alle sizing-variablene, men det er ikke så mange. Et kjapt søk viser at disse vil påvirkes:
nve-tag, nve-switch og nve-textarea.

authors: NVEJoel gruble knutnve

repo: NVE/Designsystem


text:
Tao Converter link is outdated

See above - page not found error

Dear @tiesus,
Did you find correct link for installing tao-converter?
Thanks.

https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/resources/tao-converter/files?version=v4.0.0_trt8.5.1.7_x86
I was able to run it with this version. We ended setting up the whole environment in docker, since the dependencies with cuda are much nicer to handle there.

Hi @tiesus , will it be possible to share your docker file for setting this up?

authors: PigBroA pragyandahal tiesus


text:
Move the download_data scripts into digits/

Data download scripts get distributed automatically with setup.py
Allows "python -m digits.download_data" - independent of installation method

(needed for upcoming v4.1 documentation update)

repo: NVIDIA/DIGITS


text:
Installation Error with mamba-ssm while Installing nemo-curator

Description:
I'm encountering an issue while attempting to install nemo-curator[all] via the command in google colab:
!pip install --extra-index-url https://pypi.nvidia.com nemo-curator[all]

The error log indicates that the build process for the mamba-ssm package fails, resulting in the following message:
ERROR: Failed to build installable wheels for some pyproject.toml based projects (mamba-ssm)

Error Details:

The installation completes for some packages but fails when building the wheel for mamba-ssm.
The error originates from a subprocess, and it does not seem to be a pip issue.

Solution:
I was able to resolve the issue by manually downloading and installing the specific .whl file for mamba-ssm. Below are the working commands:
!wget https://github.com/state-spaces/mamba/releases/download/v2.2.2/mamba_ssm-2.2.2+cu118torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
!pip install mamba_ssm-2.2.2+cu118torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl

These commands allow for a successful installation of the mamba-ssm package.
Environment:

Python version: 3.10
OS: Linux (x86_64)
CUDA version: cu118 (for torch 2.1)

Additional Information:
Would appreciate any updates or recommendations regarding the installation process for mamba-ssm in the nemo-curator package. This solution might be useful for others facing the same issue.

I have same error

try this
!wget https://github.com/state-spaces/mamba/releases/download/v2.2.2/mamba_ssm-2.2.2+cu118torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
!pip install mamba_ssm-2.2.2+cu118torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
before
!pip install --extra-index-url https://pypi.nvidia.com nemo-curator[all]
will work .

Thanks for raising @piyushkashyap07. @ryantwolf looks like this is related to Nemo-toolkit 2.0 which was fixed in #372 but still impacts the existing release. Perhaps we need to patch release 0.6.1 pinning nemo-toolkit<2?

Hi, should now be resolved with the latest NeMo Curator release 0.5.1. Thanks!

authors: Eom-taeseon ayushdg piyushkashyap07 sarahyurick

repo: NVIDIA/NeMo-Curator


text:
Update nemotron_340b.yaml

Update the recipe arguments to achieve similar MFU of paper https://arxiv.org/pdf/2406.11704

@dimapihtar could you PTAL @ this PR ?

authors: pablo-garay zhenghax

repo: NVIDIA/NeMo-Framework-Launcher


text:
DeviceRadixSort::SortPairs hangs if unsigned numItems is close to 2^32

The following program hangs for unsigned int number of items close to 2^32. (Did not finish after 30 minutes). The same numbers with size_t numItems works.
//nvcc -O3 -arch=sm_70 -Icub-1.16.0 -Ithrust-1.16.0 --extended-lambda main.cu -o main

#include <iostream>
#include <cassert>
#include <algorithm>
#include <numeric>
#include <limits>

#include <thrust/sort.h>
#include <thrust/execution_policy.h>
#include <thrust/version.h>

#include <cub/cub.cuh>
#include <cub/version.cuh>

int main(){

    std::cout << "cub version " << CUB_VERSION << "\n";
    std::cout << "thrust version " << THRUST_VERSION << "\n";

    cudaError_t status = cudaSuccess;

    //int numKeys = std::numeric_limits<int>::max(); //OK
    //unsigned int numKeys = static_cast<unsigned int>(std::numeric_limits<int>::max()) + 42; //OK
    //unsigned int numKeys = std::numeric_limits<unsigned int>::max() - 42; // hangs
    //unsigned int numKeys = std::numeric_limits<unsigned int>::max() - 1; // hangs
    unsigned int numKeys = std::numeric_limits<unsigned int>::max(); // hangs
    //std::size_t numKeys = std::numeric_limits<unsigned int>::max(); //OK
    //std::size_t numKeys = static_cast<std::size_t>(std::numeric_limits<unsigned int>::max()) + 42; // OK
    //std::size_t numKeys = 4737342875ull; // OK
    auto numValues = numKeys;

    std::cout << "num items: " << numKeys << "\n";

    std::cout << "allocate buffers\n";

    std::size_t* keysA;
    status = cudaMallocManaged(&keysA, sizeof(std::size_t) * numKeys);
    assert(status == cudaSuccess);

    std::size_t* keysB;
    status = cudaMallocManaged(&keysB, sizeof(std::size_t) * numKeys);
    assert(status == cudaSuccess);

    std::size_t* valuesA;
    status = cudaMallocManaged(&valuesA, sizeof(std::size_t) * numValues);
    assert(status == cudaSuccess);

    std::size_t* valuesB;
    status = cudaMallocManaged(&valuesB, sizeof(std::size_t) * numValues);
    assert(status == cudaSuccess);

    std::cout << "load data\n";
    std::iota(keysA, keysA + numKeys, std::size_t(0));
    std::reverse(keysA, keysA + numKeys);
    std::iota(valuesA, valuesA + numValues, std::size_t(0));

    cub::DoubleBuffer<std::size_t> d_keys_dbl{keysA, keysB};
    cub::DoubleBuffer<std::size_t> d_values_dbl{valuesA, valuesB};

    std::cout << "allocate cub temp\n";

    std::size_t tempbytes = 0;
    status = cub::DeviceRadixSort::SortPairs(
        nullptr,
        tempbytes,
        d_keys_dbl,
        d_values_dbl,
        numKeys,
        0,
        sizeof(std::size_t) * 8
    );
    assert(status == cudaSuccess);
    void* cubtemp;
    status = cudaMallocManaged(&cubtemp, tempbytes);
    assert(status == cudaSuccess);

    std::cout << "Sorting\n";
    status = cub::DeviceRadixSort::SortPairs(
        cubtemp,
        tempbytes,
        d_keys_dbl,
        d_values_dbl,
        numKeys,
        0,
        sizeof(std::size_t) * 8
    );
    assert(status == cudaSuccess);

    status = cudaFree(cubtemp);
    assert(status == cudaSuccess);

    std::cout << "Checking\n";

    for(std::size_t i = 0; i < numKeys; i++){
        const std::size_t expectedKey = i;
        const std::size_t expectedValue = numKeys - 1 - i;

        assert(d_keys_dbl.Current()[i] == expectedKey);
        if(d_values_dbl.Current()[i] != expectedValue){
            std::cerr << "error i = " << i
                << " key = " << expectedKey
                << " expectedValue = " << expectedValue
                << " observed value = " << d_values_dbl.Current()[i] << "\n";
        }
        assert(d_values_dbl.Current()[i] == expectedValue);
    }
    std::cout << "OK\n";

    cudaDeviceReset();
}

The problem seems to be an infinite loop in agent_radix_sort_histogram. The tile_offset variable wraps around in the last portion.
https://github.com/NVIDIA/cub/blob/a7e3495a87607e877b86aadc20c42c624957a4ea/cub/agent/agent_radix_sort_histogram.cuh#L241
https://github.com/NVIDIA/cub/blob/a7e3495a87607e877b86aadc20c42c624957a4ea/cub/agent/agent_radix_sort_histogram.cuh#L242

@fkallen Thanks for catching this!
I've just opened https://github.com/NVIDIA/cub/pull/468 to fix this issue. @fkallen Could you check whether the fix works for you? If yes, I'll add some more tests to the pull request, and then it should be ready for review.

@canonizer It works with your PR.

authors: canonizer fkallen

repo: NVIDIA/cub


text:
Hi, I ask a question again

Hi, let me ask a newbie question. You have created a Metropolis algorithm based on GPU, and it works quite good. However, for statistical physics, different dynamics have a different meaning for the same model, such that a Metropolis algorithm of Ising model just represents the Metropolis dynamics of Ising model. There are still different dynamics for Ising model, such as Wolff dynamics of Ising model or Kawasaki dynamics of conserved-order-parameter Ising model and so on. Do you have some plans to create GPU programmes for these dynamics for the user who studies statistical physics?
Best wishes

Hi @liufengyinxue,
Glad to hear you are having success using this code. We do not currently have plans to extend these programs beyond the Ising model currently implemented, as the intent of this repository is more as an example, rather than an ongoing project. Hopefully you can use what is here as a starting point to implement your own version with the dynamics you need.

authors: liufengyinxue romerojosh

repo: NVIDIA/ising-gpu


text:
crdutil: Add feature to also apply single files

This PR adds a feature to apply single yaml files, not only complete directories.

Pull Request Test Coverage Report for Build 12006963212
Details

0 of 17 (0.0%)  changed or added relevant lines in 1 file are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.5%) to 61.613%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/crdutil/crdutil.go
0
17
0.0%

Files with Coverage Reduction
New Missed Lines
%

pkg/crdutil/crdutil.go
1
31.62%

Totals

Change from base Build 11928826672:
-0.5%

Covered Lines:
1085

Relevant Lines:
1761

💛  - Coveralls

Thanks for the extensive review @shivamerla, I should have handled all findings in this diff.

Tanks for the extensive review @shivamerla. I should have implemented all findings in this diff.

authors: coveralls tobiasgiese


text:
[DRAFT] move xgboost dep to dmlc xgb 1.6.0 rc1

Please Note that, the CV is not working and the xgboost-pspark is not working.

Did you pass build locally? I got it failed as below:
https://blossom.nvidia.com/sw-gpu-spark-jenkins/view/Rapids-examples/job/examples_xgb_build_nightly/244/console
21:11:12  [INFO] /home/jenkins/agent/workspace/examples_xgb_build_nightly/examples/Spark-ETL+XGBoost/agaricus/scala/src:-1: info: compiling
21:11:12  [INFO] Compiling 2 source files to /home/jenkins/agent/workspace/examples_xgb_build_nightly/examples/Spark-ETL+XGBoost/agaricus/target/classes at 1649769071974
21:11:14  [ERROR] /home/jenkins/agent/workspace/examples_xgb_build_nightly/examples/Spark-ETL+XGBoost/agaricus/scala/src/com/nvidia/spark/examples/agaricus/GPUMain.scala:79: error: value setFeaturesCols is not a member of ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier
21:11:14  [INFO] possible cause: maybe a semicolon is missing before `value setFeaturesCols'?
21:11:14  [INFO]         .setFeaturesCols(featureCols)
21:11:14  [INFO]          ^
21:11:14  [ERROR] /home/jenkins/agent/workspace/examples_xgb_build_nightly/examples/Spark-ETL+XGBoost/agaricus/scala/src/com/nvidia/spark/examples/agaricus/GPUMain.scala:87: error: value write is not a member of Any
21:11:14  [INFO]         if (xgboostArgs.isOverwrite) model.write.overwrite().save(path) else model.save(path))
21:11:14  [INFO]                                            ^
21:11:14  [ERROR] /home/jenkins/agent/workspace/examples_xgb_build_nightly/examples/Spark-ETL+XGBoost/agaricus/scala/src/com/nvidia/spark/examples/agaricus/GPUMain.scala:87: error: value save is not a member of Any
21:11:14  [INFO]         if (xgboostArgs.isOverwrite) model.write.overwrite().save(path) else model.save(path))
21:11:14  [INFO]                                                                                ^
21:11:14  [ERROR] /home/jenkins/agent/workspace/examples_xgb_build_nightly/examples/Spark-ETL+XGBoost/agaricus/scala/src/com/nvidia/spark/examples/agaricus/GPUMain.scala:97: error: value transform is not a member of Any
21:11:14  [INFO]         val ret = xgbClassificationModel.transform(datasets(2).get).cache()
21:11:14  [INFO]                                          ^
21:11:14  [ERROR] /home/jenkins/agent/workspace/examples_xgb_build_nightly/examples/Spark-ETL+XGBoost/agaricus/scala/src/com/nvidia/spark/examples/agaricus/GPUMain.scala:104: error: value select is not a member of Any
21:11:14  [INFO]         results.select(labelName, "rawPrediction", "probability", "prediction")
21:11:14  [INFO]                 ^
21:11:14  [ERROR] /home/jenkins/agent/workspace/examples_xgb_build_nightly/examples/Spark-ETL+XGBoost/agaricus/scala/src/com/nvidia/spark/examples/agaricus/GPUMain.scala:106: error: value show is not a member of Any
21:11:14  [INFO]       results.show(xgboostArgs.numRows)
21:11:14  [INFO]               ^
21:11:14  [ERROR] /home/jenkins/agent/workspace/examples_xgb_build_nightly/examples/Spark-ETL+XGBoost/agaricus/scala/src/com/nvidia/spark/examples/agaricus/GPUMain.scala:110: error: type mismatch;
21:11:14  [INFO]  found   : Any
21:11:14  [INFO]  required: org.apache.spark.sql.Dataset[_]
21:11:14  [INFO]       evaluator.evaluate(results) match {
21:11:14  [INFO]                          ^
21:11:14  [ERROR] 7 errors found
21:11:14  [INFO] ------------------------------------------------------------------------
21:11:14  [INFO] Reactor Summary for sample_xgboost_apps 0.2.2-SNAPSHOT:
21:11:14  [INFO]
21:11:14  [INFO] sample_xgboost_apps ................................ SUCCESS [ 38.305 s]
21:11:14  [INFO] spark_examples_utility ............................. SUCCESS [  7.353 s]
21:11:14  [INFO] spark_examples_agaricus ............................ FAILURE [  2.587 s]

verified PASS: http://spark-gashen:9090/job/example_spark3.x_integration/883/

repo: NVIDIA/spark-rapids-examples


text:
[BUG] test_parquet_check_schema_compatibility failed in databricks runtimes

Describe the bug
related to https://github.com/NVIDIA/spark-rapids/pull/5434
[2022-05-13T04:29:13.200Z] FAILED ../../src/main/python/parquet_test.py::test_parquet_check_schema_compatibility

[2022-05-13T04:29:12.934Z] ___________________ test_parquet_check_schema_compatibility ____________________
[2022-05-13T04:29:12.934Z] [gw2] linux -- Python 3.8.13 /databricks/conda/envs/cudf-udf/bin/python
[2022-05-13T04:29:12.934Z]
[2022-05-13T04:29:12.934Z] spark_tmp_path = '/tmp/pyspark_tests//0513-023846-8xru1kk3-10-2-128-4-gw2-5446-2128719059/'
[2022-05-13T04:29:12.934Z]
[2022-05-13T04:29:12.934Z]     def test_parquet_check_schema_compatibility(spark_tmp_path):
[2022-05-13T04:29:12.934Z]         data_path = spark_tmp_path + '/PARQUET_DATA'
[2022-05-13T04:29:12.934Z]         gen_list = [('int', int_gen), ('long', long_gen), ('dec32', decimal_gen_32bit)]
[2022-05-13T04:29:12.934Z]         with_cpu_session(lambda spark: gen_df(spark, gen_list).coalesce(1).write.parquet(data_path))
[2022-05-13T04:29:12.934Z]
[2022-05-13T04:29:12.934Z]         read_int_as_long = StructType(
[2022-05-13T04:29:12.934Z]             [StructField('long', LongType()), StructField('int', LongType())])
[2022-05-13T04:29:12.934Z] >       assert_gpu_and_cpu_error(
[2022-05-13T04:29:12.934Z]             lambda spark: spark.read.schema(read_int_as_long).parquet(data_path).collect(),
[2022-05-13T04:29:12.934Z]             conf={},
[2022-05-13T04:29:12.934Z]             error_message='Parquet column cannot be converted in')
[2022-05-13T04:29:12.934Z]
[2022-05-13T04:29:12.934Z] ../../src/main/python/parquet_test.py:878:
[2022-05-13T04:29:12.934Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2022-05-13T04:29:12.934Z] ../../src/main/python/asserts.py:572: in assert_gpu_and_cpu_error
[2022-05-13T04:29:12.934Z]     assert_py4j_exception(lambda: with_cpu_session(df_fun, conf), error_message)
[2022-05-13T04:29:12.934Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2022-05-13T04:29:12.934Z]
[2022-05-13T04:29:12.934Z] func = <function assert_gpu_and_cpu_error.<locals>.<lambda> at 0x7f37a5e7cc10>
[2022-05-13T04:29:12.934Z] error_message = 'Parquet column cannot be converted in'
[2022-05-13T04:29:12.934Z]
[2022-05-13T04:29:12.934Z]     def assert_py4j_exception(func, error_message):
[2022-05-13T04:29:12.934Z]         """
[2022-05-13T04:29:12.934Z]         Assert that a specific Java exception is thrown
[2022-05-13T04:29:12.934Z]         :param func: a function to be verified
[2022-05-13T04:29:12.934Z]         :param error_message: a string such as the one produce by java.lang.Exception.toString
[2022-05-13T04:29:12.934Z]         :return: Assertion failure if no exception matching error_message has occurred.
[2022-05-13T04:29:12.934Z]         """
[2022-05-13T04:29:12.934Z]         with pytest.raises(Py4JJavaError) as py4jError:
[2022-05-13T04:29:12.934Z]             func()
[2022-05-13T04:29:12.934Z] >       assert error_message in str(py4jError.value.java_exception)
[2022-05-13T04:29:12.934Z] E       AssertionError
[2022-05-13T04:29:12.934Z]
[2022-05-13T04:29:12.934Z] ../../src/main/python/asserts.py:561: AssertionError
[2022-05-13T04:29:12.934Z] ----------------------------- Captured stderr call -----------------------------
[2022-05-13T04:29:12.934Z] 22/05/13 04:03:47 ERROR FileScanRDD: Error while reading file file:/tmp/pyspark_tests/0513-023846-8xru1kk3-10-2-128-4-gw2-5446-2128719059/PARQUET_DATA/part-00000-tid-4751444801957620801-e47b08e7-db20-487f-8980-d7e31862c6ae-31197-1-c000.snappy.parquet. Parquet column cannot be converted. Column: [int], Expected: LongType, Found: INT32
[2022-05-13T04:29:12.934Z] 22/05/13 04:03:47 ERROR FileScanRDD: Error in async I/O
[2022-05-13T04:29:12.934Z] org.apache.spark.SparkException: Exception thrown in awaitResult:
[2022-05-13T04:29:12.934Z] 	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:428)
[2022-05-13T04:29:12.934Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:449)
[2022-05-13T04:29:12.934Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.$anonfun$hasNext$1(FileScanRDD.scala:302)
[2022-05-13T04:29:12.934Z] 	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
[2022-05-13T04:29:12.934Z] 	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:297)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:845)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:757)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.encodeUnsafeRows(UnsafeRowBatchUtils.scala:80)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.execution.collect.Collector.$anonfun$processFunc$1(Collector.scala:178)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
[2022-05-13T04:29:12.935Z] 	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
[2022-05-13T04:29:12.935Z] 	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.scheduler.Task.doRunTask(Task.scala:150)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:119)
[2022-05-13T04:29:12.935Z] 	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.scheduler.Task.run(Task.scala:91)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:813)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1646)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:816)
[2022-05-13T04:29:12.935Z] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[2022-05-13T04:29:12.935Z] 	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:672)
[2022-05-13T04:29:12.935Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2022-05-13T04:29:12.935Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2022-05-13T04:29:12.935Z] 	at java.lang.Thread.run(Thread.java:748)
[2022-05-13T04:29:12.935Z] Caused by: com.databricks.sql.io.FileReadException: Error while reading file file:/tmp/pyspark_tests/0513-023846-8xru1kk3-10-2-128-4-gw2-5446-2128719059/PARQUET_DATA/part-00000-tid-4751444801957620801-e47b08e7-db20-487f-8980-d7e31862c6ae-31197-1-c000.snappy.parquet. Parquet column cannot be converted. Column: [int], Expected: LongType, Found: INT32
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.logFileNameAndThrow(FileScanRDD.scala:391)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:361)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.$anonfun$prepareNextFile$1(FileScanRDD.scala:564)
[2022-05-13T04:29:12.935Z] 	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[2022-05-13T04:29:12.935Z] 	at scala.util.Success.$anonfun$map$1(Try.scala:255)
[2022-05-13T04:29:12.935Z] 	at scala.util.Success.map(Try.scala:213)
[2022-05-13T04:29:12.935Z] 	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[2022-05-13T04:29:12.935Z] 	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[2022-05-13T04:29:12.935Z] 	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[2022-05-13T04:29:12.935Z] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.$anonfun$run$1(SparkThreadLocalForwardingThreadPoolExecutor.scala:104)
[2022-05-13T04:29:12.935Z] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:68)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured$(SparkThreadLocalForwardingThreadPoolExecutor.scala:54)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:101)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.run(SparkThreadLocalForwardingThreadPoolExecutor.scala:104)
[2022-05-13T04:29:12.935Z] 	... 3 more
[2022-05-13T04:29:12.935Z] Caused by: org.apache.spark.sql.execution.datasources.SchemaColumnConvertNotSupportedException
[2022-05-13T04:29:12.935Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.constructConvertNotSupportedException(VectorizedColumnReader.java:448)
[2022-05-13T04:29:12.935Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.readIntBatch(VectorizedColumnReader.java:693)
[2022-05-13T04:29:12.935Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:391)
[2022-05-13T04:29:12.935Z] 	at com.databricks.sql.io.parquet.DatabricksVectorizedParquetRecordReader.nextBatch(DatabricksVectorizedParquetRecordReader.java:480)
[2022-05-13T04:29:12.935Z] 	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:190)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:335)
[2022-05-13T04:29:12.936Z] 	... 18 more
[2022-05-13T04:29:12.936Z] 22/05/13 04:03:47 ERROR Executor: Exception in task 0.0 in stage 11596.0 (TID 31198)
[2022-05-13T04:29:12.936Z] com.databricks.sql.io.FileReadException: Error while reading file file:/tmp/pyspark_tests/0513-023846-8xru1kk3-10-2-128-4-gw2-5446-2128719059/PARQUET_DATA/part-00000-tid-4751444801957620801-e47b08e7-db20-487f-8980-d7e31862c6ae-31197-1-c000.snappy.parquet. Parquet column cannot be converted. Column: [int], Expected: LongType, Found: INT32
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.logFileNameAndThrow(FileScanRDD.scala:391)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:361)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.$anonfun$prepareNextFile$1(FileScanRDD.scala:564)
[2022-05-13T04:29:12.936Z] 	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[2022-05-13T04:29:12.936Z] 	at scala.util.Success.$anonfun$map$1(Try.scala:255)
[2022-05-13T04:29:12.936Z] 	at scala.util.Success.map(Try.scala:213)
[2022-05-13T04:29:12.936Z] 	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[2022-05-13T04:29:12.936Z] 	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[2022-05-13T04:29:12.936Z] 	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[2022-05-13T04:29:12.936Z] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.$anonfun$run$1(SparkThreadLocalForwardingThreadPoolExecutor.scala:104)
[2022-05-13T04:29:12.936Z] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:68)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured$(SparkThreadLocalForwardingThreadPoolExecutor.scala:54)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:101)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.run(SparkThreadLocalForwardingThreadPoolExecutor.scala:104)
[2022-05-13T04:29:12.936Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2022-05-13T04:29:12.936Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2022-05-13T04:29:12.936Z] 	at java.lang.Thread.run(Thread.java:748)
[2022-05-13T04:29:12.936Z] Caused by: org.apache.spark.sql.execution.datasources.SchemaColumnConvertNotSupportedException
[2022-05-13T04:29:12.936Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.constructConvertNotSupportedException(VectorizedColumnReader.java:448)
[2022-05-13T04:29:12.936Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.readIntBatch(VectorizedColumnReader.java:693)
[2022-05-13T04:29:12.936Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:391)
[2022-05-13T04:29:12.936Z] 	at com.databricks.sql.io.parquet.DatabricksVectorizedParquetRecordReader.nextBatch(DatabricksVectorizedParquetRecordReader.java:480)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:190)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:335)
[2022-05-13T04:29:12.936Z] 	... 18 more
[2022-05-13T04:29:12.936Z] 22/05/13 04:03:47 ERROR FileScanRDD: Error in async I/O
[2022-05-13T04:29:12.936Z] com.databricks.sql.io.FileReadException: Error while reading file file:/tmp/pyspark_tests/0513-023846-8xru1kk3-10-2-128-4-gw2-5446-2128719059/PARQUET_DATA/part-00000-tid-4751444801957620801-e47b08e7-db20-487f-8980-d7e31862c6ae-31197-1-c000.snappy.parquet. Parquet column cannot be converted. Column: [int], Expected: LongType, Found: INT32
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.logFileNameAndThrow(FileScanRDD.scala:391)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:361)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.$anonfun$prepareNextFile$1(FileScanRDD.scala:564)
[2022-05-13T04:29:12.936Z] 	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[2022-05-13T04:29:12.936Z] 	at scala.util.Success.$anonfun$map$1(Try.scala:255)
[2022-05-13T04:29:12.936Z] 	at scala.util.Success.map(Try.scala:213)
[2022-05-13T04:29:12.936Z] 	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[2022-05-13T04:29:12.936Z] 	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[2022-05-13T04:29:12.936Z] 	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[2022-05-13T04:29:12.936Z] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[2022-05-13T04:29:12.936Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.$anonfun$run$1(SparkThreadLocalForwardingThreadPoolExecutor.scala:104)
[2022-05-13T04:29:12.936Z] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:68)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured$(SparkThreadLocalForwardingThreadPoolExecutor.scala:54)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:101)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.run(SparkThreadLocalForwardingThreadPoolExecutor.scala:104)
[2022-05-13T04:29:12.937Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2022-05-13T04:29:12.937Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2022-05-13T04:29:12.937Z] 	at java.lang.Thread.run(Thread.java:748)
[2022-05-13T04:29:12.937Z] Caused by: org.apache.spark.sql.execution.datasources.SchemaColumnConvertNotSupportedException
[2022-05-13T04:29:12.937Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.constructConvertNotSupportedException(VectorizedColumnReader.java:448)
[2022-05-13T04:29:12.937Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.readIntBatch(VectorizedColumnReader.java:693)
[2022-05-13T04:29:12.937Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:391)
[2022-05-13T04:29:12.937Z] 	at com.databricks.sql.io.parquet.DatabricksVectorizedParquetRecordReader.nextBatch(DatabricksVectorizedParquetRecordReader.java:480)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:190)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:335)
[2022-05-13T04:29:12.937Z] 	... 18 more
[2022-05-13T04:29:12.937Z] 22/05/13 04:03:47 WARN TaskSetManager: Lost task 0.0 in stage 11596.0 (TID 31198) (10.2.128.4 executor driver): com.databricks.sql.io.FileReadException: Error while reading file file:/tmp/pyspark_tests/0513-023846-8xru1kk3-10-2-128-4-gw2-5446-2128719059/PARQUET_DATA/part-00000-tid-4751444801957620801-e47b08e7-db20-487f-8980-d7e31862c6ae-31197-1-c000.snappy.parquet. Parquet column cannot be converted. Column: [int], Expected: LongType, Found: INT32
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.logFileNameAndThrow(FileScanRDD.scala:391)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:361)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.$anonfun$prepareNextFile$1(FileScanRDD.scala:564)
[2022-05-13T04:29:12.937Z] 	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[2022-05-13T04:29:12.937Z] 	at scala.util.Success.$anonfun$map$1(Try.scala:255)
[2022-05-13T04:29:12.937Z] 	at scala.util.Success.map(Try.scala:213)
[2022-05-13T04:29:12.937Z] 	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[2022-05-13T04:29:12.937Z] 	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[2022-05-13T04:29:12.937Z] 	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[2022-05-13T04:29:12.937Z] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.$anonfun$run$1(SparkThreadLocalForwardingThreadPoolExecutor.scala:104)
[2022-05-13T04:29:12.937Z] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:68)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured$(SparkThreadLocalForwardingThreadPoolExecutor.scala:54)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:101)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.run(SparkThreadLocalForwardingThreadPoolExecutor.scala:104)
[2022-05-13T04:29:12.937Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2022-05-13T04:29:12.937Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2022-05-13T04:29:12.937Z] 	at java.lang.Thread.run(Thread.java:748)
[2022-05-13T04:29:12.937Z] Caused by: org.apache.spark.sql.execution.datasources.SchemaColumnConvertNotSupportedException
[2022-05-13T04:29:12.937Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.constructConvertNotSupportedException(VectorizedColumnReader.java:448)
[2022-05-13T04:29:12.937Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.readIntBatch(VectorizedColumnReader.java:693)
[2022-05-13T04:29:12.937Z] 	at com.databricks.sql.io.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:391)
[2022-05-13T04:29:12.937Z] 	at com.databricks.sql.io.parquet.DatabricksVectorizedParquetRecordReader.nextBatch(DatabricksVectorizedParquetRecordReader.java:480)
[2022-05-13T04:29:12.937Z] 	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:190)
[2022-05-13T04:29:12.938Z] 	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
[2022-05-13T04:29:12.938Z] 	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:335)
[2022-05-13T04:29:12.938Z] 	... 18 more

Fixed in #5494.

authors: pxLi res-life

repo: NVIDIA/spark-rapids


text:
[BUG] Non-deterministic query result corruption when RAPIDS shuffle manager is enabled

Describe the bug
A query may produce incorrect results non-deterministically when the shuffle manager is enabled.
Steps/Code to reproduce bug
Start pyspark with 2 executors with a local standalone backend or an equivalent local-cluster
$SPARK_HOME/bin/pyspark \
  --driver-class-path $PWD/dist/target/rapids-4-spark_2.12-22.06.0-SNAPSHOT-cuda11.jar  \
  --conf spark.executor.extraClassPath=$PWD/dist/target/rapids-4-spark_2.12-22.06.0-SNAPSHOT-cuda11.jar  \
  --conf spark.plugins=com.nvidia.spark.SQLPlugin  \
  --conf spark.rapids.sql.explain=ALL \
  --conf spark.shuffle.manager=com.nvidia.spark.rapids.spark321.RapidsShuffleManager \
  --conf spark.shuffle.service.enabled=false  \
  --conf spark.dynamicAllocation.enabled=false  \
  --conf spark.executorEnv.UCX_ERROR_SIGNALS=  \
  --conf spark.executorEnv.UCX_MEMTYPE_CACHE=n \
  --conf spark.rapids.memory.gpu.minAllocFraction=0   \
  --conf spark.rapids.memory.gpu.allocFraction=0.2 \
  --conf spark.rapids.shuffle.enabled=false  \
  --master local-cluster[2,1,1200]

if running with a full local standalone backend use
--executor-cores=2 --total-executor-cores=4 --master spark://$(hostname):7077

instead of --master local-cluster[2,1024,1200]
Materialize a simple dataframe in Parquet (same with avro)
>>> spark.createDataFrame([ [1], [10] ], 'a int').write.format('parquet').save(path='/tmp/df.parquet', mode='overwrite')

Repeatedly execute the following query
>>> spark.read.format('parquet').load('/tmp/df.parquet').selectExpr('avg(a)').collect()

and observe the  correct result [Row(avg(a)=5.5)] without any issues
Enable RAPIDS Shuffle
>>> spark.conf.set('spark.rapids.shuffle.enabled', True)

and re-execute the query many times and see the query bouncing between [Row(avg(a)=10.0)],  [Row(avg(a)=1.0)],  [Row(avg(a)=5.5)]
Expected behavior
AVG over [1, 10] should be a deterministic query regardless of partitioning and format
Environment details (please complete the following information)

Environment location: Standalone
Spark configuration settings related to the issue: see above

Additional context
N/A

@gerashegalov, I am trying to follow these instructions and I can't replicate it so far. I tried local-cluster mode and standalone mode. Could you provide a bit more information:

What version of ucx do you have installed (ucx_info -v). I am running 1.12.0 locally.
I see 5.5 average constantly. Do you ever see that? Or is it 1 or 10 alternating. I am wondering if the issue that you are seeing is on the write, or on the read (or both). It would be nice to get it to happen, and try to inspect the parquet file on the CPU or with the shuffle manger "off" (e.g. a read + a show to get its contents).

@abellina Interestingly this behavior has not reproduced for me yet on a bare-metal Ubuntu20.04 although it reproduces very easily on WSL2 VM Ubuntu20.04

Debugging with h/t @abellina it looks like the workaround is to set  --conf spark.executorEnv.UCX_TLS=cuda_copy,tcp

Wed Jun 15 14:41:09 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.48.03    Driver Version: 516.25       CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+

ucx debug log files for the repro run
$ $SPARK_HOME/bin/pyspark   --driver-class-path $PWD/dist/target/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar    --conf spark.executor.extraClassPath=$PWD/dist/target/rapids-4-spark_2.12-22.08.0-SNAPSHOT-cuda11.jar    --conf spark.plugins=com.nvidia.spark.SQLPlugin    --conf spark.rapids.sql.explain=ALL   --conf spark.shuffle.manager=com.nvidia.spark.rapids.spark321.RapidsShuffleManager   --conf spark.shuffle.service.enabled=false    --conf spark.dynamicAllocation.enabled=false  --conf spark.executorEnv.UCX_ERROR_SIGNALS= --conf spark.executorEnv.UCX_LOG_LEVEL=data --conf spark.executorEnv.UCX_LOG_FILE=/tmp/ucx_log_%p --conf spark.executorEnv.LD_LIBRARY_PATH=$HOME/dist/ucx_debug/lib --conf spark.executorEnv.UCX_MEMTYPE_CACHE=n   --conf spark.rapids.memory.gpu.minAllocFraction=0  --conf spark.rapids.memory.gpu.allocFraction=0.2 --master local-cluster[2,1,1200]
Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
22/06/18 01:19:08 WARN Utils: Your hostname, NV-3L4YVG3 resolves to a loopback address: 127.0.1.1; using 172.22.19.221 instead (on interface eth0)
22/06/18 01:19:08 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/06/18 01:19:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/06/18 01:19:10 WARN RapidsPluginUtils: RAPIDS Accelerator 22.08.0-SNAPSHOT using cudf 22.08.0-SNAPSHOT.
22/06/18 01:19:10 WARN RapidsPluginUtils: RAPIDS Accelerator is enabled, to disable GPU support set `spark.rapids.sql.enabled` to false.
22/06/18 01:19:10 WARN RapidsShuffleInternalManager: Rapids Shuffle Plugin enabled. Transport enabled (remote fetches will use com.nvidia.spark.rapids.shuffle.ucx.UCXShuffleTransport. To disable the RAPIDS Shuffle Manager set `spark.rapids.shuffle.enabled` to false
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.2.1
      /_/
Using Python version 3.8.13 (default, Mar 25 2022 06:04:10)
Spark context Web UI available at http://172.22.19.221:4040/
Spark context available as 'sc' (master = local-cluster[2,1,1200], app id = app-20220618011911-0000).
SparkSession available as 'spark'.

spark.read.format('parquet').load('/tmp/df.parquet').selectExpr('avg(a)').collect()
22/06/18 01:19:23 WARN GpuOverrides:
*Exec  will run on GPU
*Expression  avg(a#0) will run on GPU
*Expression  avg(a#0) will run on GPU
*Expression  avg(a#0)#2 AS avg(a)#3 will run on GPU
*Exec  will run on GPU
*Partitioning <SinglePartition$> will run on GPU
*Exec  will run on GPU
*Expression  partial_avg(a#0) will run on GPU
*Expression  avg(a#0) will run on GPU
*Exec  will run on GPU

22/06/18 01:19:23 WARN GpuOverrides:
*Exec  will run on GPU
*Expression  avg(a#0) will run on GPU
*Expression  avg(a#0) will run on GPU
*Expression  avg(a#0)#2 AS avg(a)#3 will run on GPU
*Exec  will run on GPU
*Partitioning <SinglePartition$> will run on GPU
*Exec  will run on GPU
*Expression  partial_avg(a#0) will run on GPU
*Expression  avg(a#0) will run on GPU
*Exec  will run on GPU
22/06/18 01:19:23 WARN GpuOverrides:
*Exec  will run on GPU
*Expression  avg(a#0) will run on GPU
*Expression  avg(a#0) will run on GPU
*Expression  avg(a#0)#2 AS avg(a)#3 will run on GPU
*Exec  will run on GPU
*Partitioning <SinglePartition$> will run on GPU
*Exec  will run on GPU
*Expression  partial_avg(a#0) will run on GPU
*Expression  avg(a#0) will run on GPU
*Exec  will run on GPU
22/06/18 01:19:23 WARN GpuOverrides:
*Exec  will run on GPU
*Partitioning <SinglePartition$> will run on GPU
*Exec  will run on GPU
*Expression  partial_avg(a#0) will run on GPU
*Expression  avg(a#0) will run on GPU
*Exec  will run on GPU
22/06/18 01:19:25 WARN GpuOverrides: =>                             (1 + 1) / 2]
*Exec  will run on GPU
*Expression  avg(a#0) will run on GPU
*Expression  avg(a#0) will run on GPU
*Expression  avg(a#0)#2 AS avg(a)#3 will run on GPU
22/06/18 01:19:25 WARN GpuOverrides:
*Exec  will run on GPU
*Expression  avg(a#0) will run on GPU
*Expression  avg(a#0) will run on GPU
*Expression  avg(a#0)#2 AS avg(a)#3 will run on GPU
[Row(avg(a)=1.0)]

spark.read.format('parquet').load('/tmp/df.parquet').selectExpr('avg(a)').collect()
22/06/18 01:19:27 WARN GpuOverrides:
*Exec  will run on GPU
*Expression  avg(a#31) will run on GPU
*Expression  avg(a#31) will run on GPU
*Expression  avg(a#31)#33 AS avg(a)#34 will run on GPU
*Exec  will run on GPU
*Partitioning <SinglePartition$> will run on GPU
*Exec  will run on GPU
*Expression  partial_avg(a#31) will run on GPU
*Expression  avg(a#31) will run on GPU
*Exec  will run on GPU

22/06/18 01:19:27 WARN GpuOverrides:
*Exec  will run on GPU
*Expression  avg(a#31) will run on GPU
*Expression  avg(a#31) will run on GPU
*Expression  avg(a#31)#33 AS avg(a)#34 will run on GPU
*Exec  will run on GPU
*Partitioning <SinglePartition$> will run on GPU
*Exec  will run on GPU
*Expression  partial_avg(a#31) will run on GPU
*Expression  avg(a#31) will run on GPU
*Exec  will run on GPU
22/06/18 01:19:27 WARN GpuOverrides:
*Exec  will run on GPU
*Expression  avg(a#31) will run on GPU
*Expression  avg(a#31) will run on GPU
*Expression  avg(a#31)#33 AS avg(a)#34 will run on GPU
*Exec  will run on GPU
*Partitioning <SinglePartition$> will run on GPU
*Exec  will run on GPU
*Expression  partial_avg(a#31) will run on GPU
*Expression  avg(a#31) will run on GPU
*Exec  will run on GPU
22/06/18 01:19:27 WARN GpuOverrides:
*Exec  will run on GPU
*Partitioning <SinglePartition$> will run on GPU
*Exec  will run on GPU
*Expression  partial_avg(a#31) will run on GPU
*Expression  avg(a#31) will run on GPU
*Exec  will run on GPU
22/06/18 01:19:27 WARN GpuOverrides:
*Exec  will run on GPU
*Expression  avg(a#31) will run on GPU
*Expression  avg(a#31) will run on GPU
*Expression  avg(a#31)#33 AS avg(a)#34 will run on GPU
22/06/18 01:19:27 WARN GpuOverrides:
*Exec  will run on GPU
*Expression  avg(a#31) will run on GPU
*Expression  avg(a#31) will run on GPU
*Expression  avg(a#31)#33 AS avg(a)#34 will run on GPU
[Row(avg(a)=10.0)]

[ucx_log_sanitized.zip](https://github.com/NVIDIA/spark-rapids/files/8934760/ucx_log_sanitized.zip)

OK, thanks for the logs! With @gerashegalov's help I was able to setup WSL2 and repro this in a windows machine.
The issue is specific to cuda_ipc with wakeup, which is using events to trigger when a send/recv is finished. What looks to be happening is that we are getting a callback from UCX even though the copy is not complete, which explains why it sometimes works. This is our default mode, rather than busy waiting, since we've found it to be better for performance and we know it works in other environments. Setting: --conf spark.rapids.shuffle.ucx.useWakeup=false (turning off wakeup and relying on busy waiting), is a potential workaround for WSL2, but it's not great. This looks to be a bug in UCX or CUDA (callback functions).
I added some logging to capture the batch being received and the corresponding sent batch. This is the batch at the receiving end:
DEBUG fetched cb Table{columns=[ColumnVector{rows=1, type=FLOAT64, nullCount=Optional.empty, offHeap=(ID: 27 7f77914ebbb0)}, ColumnVector{rows=1, type=INT64, nullCount=Optional.empty, offHeap=(ID: 28 7f77914fffd0)}], cudfTable=140151513387040, rows=1}
GPU COLUMN 0 - NC: 0 DATA: DeviceMemoryBufferView{address=0x800000200, length=8, id=-1} VAL: null
COLUMN 0 - FLOAT64
0 0.0
GPU COLUMN 1 - NC: 0 DATA: DeviceMemoryBufferView{address=0x800000240, length=8, id=-1} VAL: null
COLUMN 1 - INT64
0 0

But the sender sent this (before UCX send):
DEBUG to send Table{columns=[ColumnVector{rows=1, type=FLOAT64, nullCount=Optional.empty, offHeap=(ID: 25 7f160808ea00)}, ColumnVector{rows=1, type=INT64, nullCount=Optional.empty, offHeap=(ID: 26 7f160808eb70)}], cudfTable=139732600243792, rows=1}
GPU COLUMN 0 - NC: 0 DATA: DeviceMemoryBufferView{address=0x800000000, length=8, id=-1} VAL: null
COLUMN 0 - FLOAT64
0 10.0
GPU COLUMN 1 - NC: 0 DATA: DeviceMemoryBufferView{address=0x800000040, length=8, id=-1} VAL: null
COLUMN 1 - INT64
0 1

I believe the next step is to see if the above explanation makes sense to @Akshay-Venkatesh, then we may need to help repro at a lower level or help test a fix.

Removing P1 and removing from 22.08 since the issue only occurs in WSL2 (which we do not support).

@Akshay-Venkatesh we haven't been able to reproduce the issue on available bare-metal Ubuntu environments. So it may be related to either virtualization aspect of WSL2 or to the Runtime/Driver combination aspect of WSL2. We statically link 11.5 CUDA Runtime.

authors: abellina gerashegalov sameerz


text:
[BUG] spark.rapids.sql.exec.CollectLimitExec=true can mess up the CSV header row

If we enable spark.rapids.sql.exec.CollectLimitExec=true on a 2 nodes cluster, the CSV with header may be messed up.
For example, let's use this example csv file:
wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Public/data/news_category_train.csv

The format of this csv file is like this:
category,description
Business," Short sellers, Wall Street's dwindling band of ultra cynics, are seeing green again."
Business," Private investment firm Carlyle Group, which has a reputation for making well timed and occasionally controversial plays in the defense industry, has quietly placed its bets on another part of the market."
Business, Soaring crude prices plus worries about the economy and the outlook for earnings are expected to hang over the stock market next week during the depth of the summer doldrums.

I can reproduce the issue in both databricks and dataproc.
Here is the minimum repro on dataproc:

After a 2-nodes Dataproc cluster is ready, ssh to master node

gcloud compute ssh $CLUSTER_NAME-w-0 --project=rapids-spark --zone=$ZONE
wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Public/data/news_category_train.csv
hadoop fs -put news_category_train.csv /tmp/

in spark-shell

spark.conf.set("spark.rapids.sql.exec.CollectLimitExec",true)
spark.read.option("header", true).csv("/tmp/news_category_train.csv").show(5, truncate=50)

run above command couple of times, some times it will show result like:

scala> spark.read.option("header", true).csv("/tmp/news_category_train.csv").show(5, truncate=50)
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|Sci/Tech|Scot Wingo, author of eBay Strategies: 10 Proven Methods to Maximize Your eBay Business, will answer reader questions about the online marketplace. Wingo is president and chief executive of ChannelAdvisor, an eBay consignment franchise.|
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|Business|                                                                                Short sellers, Wall Street's dwindling band of...|
|Business|                                                                                Private investment firm Carlyle Group, which h...|

Sometimes it will show correct result:
scala> spark.read.option("header", true).csv("/tmp/news_category_train.csv").show(5, truncate=50)
+--------+--------------------------------------------------+
|category|                                       description|
+--------+--------------------------------------------------+
|Business| Short sellers, Wall Street's dwindling band of...|
|Business| Private investment firm Carlyle Group, which h...|
|Business| Soaring crude prices plus worries about the ec...|
|Business| Authorities have halted oil export flows from ...|
|Business| Tearaway world oil prices, toppling records an...|
+--------+--------------------------------------------------+
only showing top 5 rows

Env:
I can reproduce using latest 22.10 snapshot and also 22.06GA jar

Does "2 nodes cluster" mean two executors or two workers ?

Actually this is not a CSV issue, but an issue of the GPU version of CollectLlimitExec.
The GPU version of CollectLimitExec will not always return the head line as the first row due to a shuffle inside it, but Spark treats the returned single row as the CSV header line when inferring the CSV schema.
Details can be found in this early issue https://github.com/NVIDIA/spark-rapids/issues/882, especially the comment https://github.com/NVIDIA/spark-rapids/issues/882#issuecomment-702343763 .

Move out of 22.12 target.

authors: GaryShen2008 firestarman viadea


text:
Replace ThreadPoolExecutor creation with ThreadUtils API [databricks]

This PR replaces the manual thread pool creation with the TrampolieUtil API which uses the org.apache.spark.util.ThreadUtils under the hood.
fixes #10759

@jlowe I have addressed your concern. PTAL

build

authors: jlowe razajafri


text:
Widen type promotion for decimals with larger scale in Parquet Read [databricks]

This PR contributes to https://github.com/NVIDIA/spark-rapids/issues/11433 and contributes to https://github.com/NVIDIA/spark-rapids/issues/11512
This PR supports additional type promotion to decimals with larger precision and scale.
As long as the precision increases by at least as much as the scale, the decimal values can be promoted without loss of precision.
A similar change is added in Apache Spark-4.0 version - https://github.com/apache/spark/pull/44513
Currently, the code throws an Exception if the scale of read schema is not as same as the schema that was written for all versions previous to Spark-4.0 on CPU. This fix is available for all versions in spark-rapids.
We have removed separate checks for the decimal  if they can be read as int, long and byte_array and consolidated into one function canReadAsDecimal. Added integration test to verify that the conditions of the type promotions are met.

build

build

@revans2 @mythrocks  I have addressed the review comments. Reverted the formatting changes. PTAL.

build

build


text:
BenchmarkRunner - Include query name in JSON summary filename

This fixes https://github.com/NVIDIA/spark-rapids/issues/1650.
Signed-off-by: Niranjan Artal nartal@nvidia.com

build


text:
Re-enable empty repetition near end-of-line anchor for rlike, regexp_extract and regexp_replace

Fixes #5659.
This enables regular expressions with empty repetitions adjacent to end-of-line anchors in every regular expression SQL function but split(), a use case that had been working before but had been excluded due to other edge cases that were not functioning.  This does this by modifying to checks to narrowly allow this particular case.

build

build


text:
Added missing include of cstdlib header file for EXIT_SUCCESS and EXIT_FAILURE

The build time check STDEXEC_FOUND_IO_URING at https://github.com/NVIDIA/stdexec/blob/main/CMakeLists.txt#L342 is failing on some systems due to undeclared macros EXIT_SUCCESS and EXIT_FAILURE.
See e.g. https://godbolt.org/z/Y9EaqrWja
The macros are defined in the standard header <cstdlib>.
However, the check is not including this header.
See e.g. https://en.cppreference.com/w/cpp/utility/program/EXIT_status
Resolves #1291

Thanks!

authors: ericniebler hghdev

repo: NVIDIA/stdexec


text:
Can't generate

Hello. I use the last commit and the last checkpoint, but when I try generate wav I take this error. I use pytorch 1.0, CUDA 8, python 3.6

+1

repo: NVIDIA/waveglow


text:
[Runtime Bug]: Minecraft Story Mode displays white screen when ran

Describe the bug
On opening with rtx remix it seems to run normally exept a white screen, I've tried using dxvk.conf with "d3d9.shaderModel = 1" but it makes it crash with this error
Could not initialize Direct3D.
Make sure you have current video drivers and
close any other applications that might be running.
You will need a video card that supports at least pixel shader version
1.2.

strangely, dxvk 2.4.1's d3d9.dll with that same dxvk.conf runs perfectly fine with no issues.
How do you reproduce the bug?
Place the RTX remix runtime in minecraft story mode's files to make the screen only show white, an make a dxvk.conf with d3d9.shaderModel = 1 to make it show the error and crash.
What is the expected behavior?
The runtime should not display a white screen(not exactly sure what it should show), and with d3d9.shaderModel = 1 it should not crash
Version
0.5.4
Logs
MinecraftStoryMode_d3d9 (remix).log
Crash dumps
(no dmp files? not sure if im missing something im extremely inexperienced)
Media
by default with rtx remix
https://github.com/user-attachments/assets/141805d1-d958-40e6-ba5b-9040dc82ef5e
These 2 images are when d3d9.shaderModel = 1 is added

(stuff behind is my wallpaper)

Interesting that mainline DXVK manages to run the game fine with the lowered shader model override. If the game supports a SM that low then I feel like Remix compatibility should be viable if not for the failed launching

Thanks for reaching out! This looks like it may be a setup issue - we suggest checking in with the Discord community to see if anyone has had success with this title. Please let us know if you have any other concerns!

Hi @EthonicDev - checking in to see if you've been able to resolve this with the community's help, or if you need further assistance. We also want to inform you of our stale ticket policy, namely that this issue will be closed as 'went stale' if it doesn't receive an update in the next 2 weeks. Thank you!

authors: EthonicDev NV-LL automata-rtx

repo: NVIDIAGameWorks/rtx-remix


text:
Remove madness

PR Type

[ ] Breaking change
[ ] Feature
[X] Patch

Brief Description
This PR is to remove the dependency of MADNESS of ParallelZone. All parallel environment initialization and finalization functions from MADNESS have been replaced by the original MPI functions.

Not In Scope

PR Checklist

[ ] Is documented
[X] Is tested
[ ] Adheres to applicable organization standards

TODOs

[ ] Task 1

🚀 [bumpr] Bumped! New version:v0.1.20 Changes:v0.1.19...v0.1.20

repo: NWChemEx/ParallelZone


text:
Question

Hello, i want to use this rat tool (not the rat) on my main computer. Is the "gui" virus?

By GUI I assume you mean the server? No it's not. It's detected as a virus because some of the client code is stored in the server. Because this program is quite powerful and can be misused its detected by most AntiViruses, you can either try to crypt your stub or add an exception to both the client and server in your antiviruses. If you are still worried the project is completely open source so feel free to check the code yourself or get someone else who can :)

By GUI I assume you mean the server? No it's not. It's detected as a virus because some of the client code is stored in the server. Because this program is quite powerful and can be misused its detected by most AntiViruses, you can either try to crypt your stub or add an exception to both the client and server in your antiviruses. If you are still worried the project is completely open source so feel free to check the code yourself or get someone else who can :)

Yes i was meaning server, thank you for answer

repo: NYAN-x-CAT/AsyncRAT-C-Sharp