text:
corrupt data when defining very large number of variables

Reported by ansley b manke on 11 Nov 2016 01:31 UTC
Ned Cokelet ran into this.  He's reading a file, reshaping the data to a new axis, defining several masks and a regridding operation, all for a dataset with 79 variables.
His script winds up defining over 1400 variables, and somewhere along the line, the internal context or memory resident information is being corrupted; the wrong data is loaded into memory.
Ned and I have reworked the script so it doesn't have all the variables defined and loaded at once. However if a limit is being reached, Ferret should detect that and report an error.
I will attach the set of scripts and the dataset that they read.  Run the script "run.jnl" to run the example.
Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/2485

Comment by steven.c.hankin on 17 Nov 2016 19:33 UTC
Unable to see the error in detail, but speculating ...
Ferret memory consists of a large contiguous array ("memory") managed as a doubly linked list of blocks.  The likely cause of the bug reported here is that somehow the management of blocks in the large memory array gets corrupted, resulting in a block that was "owned" by one variable getting overwritten by another variable.  Likely the linked lists got corrupted.  Debugging will not be fun, as the error shows up only after the lists have become very tangled and complex.
If trac ticket #2369 gets implemented the entire machinery of memory block management will disappear.  So if we are going to move forward on that ticket, it may make sense to postpone the investigation of this misbehaving script until afterwards.

Comment by ansley.b.manke on 17 Nov 2016 20:43 UTC
If I turn on "MODE DIAGNOSTIC", it seems more that the mr list is corrupted.  At some point, on loading one computed variable, an unrelated variable is being read,
yes? set mode diag
yes? go run
...
...

save/file=($2)/outtype=float/append ir_thermo_temperature_filtered
 !-> LIST/FORMAT=CDF/file=Saildrone_sd-128_data.nc/outtype=float/append ir_thermo_temperature_filtered

...  many lines of diagnostic output ...

 reading WIN-S_IN M:307 dset:   1 I:    1 1309  J: -999 -999  K: -999 -999  L: -999 -999  M:    1    1  N: -999 -999

which is one of several WIND_SPEED variables and has nothing to do with computing ir_thermo_temperature_filtered.  So it seems to me that the mr pointer info has gotten corrupted, and not the contents of memory that's been read in.
However I agree that it's well worth pursuing the new memory management first before delving into this any more.

Comment by karl.smith on 17 Nov 2016 21:46 UTC
This is starting to remind me of #2462.
The diagnostic output now looks like (from PyFerret):
yes? set mode diag
yes? use coads_climatology
yes? let a = 0; let b = 0; let c = 0
yes? can var c
yes? can var /noerr c b
yes? let b = `a`
 getgrid EX#1     C:  5 dset:   1 I:    1    1  J:    1    1  K:    1    1  L:    1    1  M:    1    1  N:    1    1
 getgrid A        C:  7 dset:   1 I:    1    1  J:    1    1  K:    1    1  L:    1    1  M:    1    1  N:    1    1
 eval    EX#1     C:  4 dset:   1 I: -999 -999  J: -999 -999  K: -999 -999  L: -999 -999  M: -999 -999  N: -999 -999
 eval    A        C:  6 dset:   0 I: -999 -999  J: -999 -999  K: -999 -999  L: -999 -999  M: -999 -999  N: -999 -999
 constan cnst     M:  2 dset:   0 I: -999 -999  J: -999 -999  K: -999 -999  L: -999 -999  M: -999 -999  N: -999 -999
 -DELETE cnst     M:  2 dset:   0 I: -999 -999  J: -999 -999  K: -999 -999  L: -999 -999  M: -999 -999  N: -999 -999
 -DELETE A        M:  2 dset:   0 I: -999 -999  J: -999 -999  K: -999 -999  L: -999 -999  M: -999 -999  N: -999 -999
 !-> DEFINE VARIABLE b = 0
yes? let a = 0
 dealloc  dynamic grid ABSTRACT        ABSTRACT  ABSTRACT  ABSTRACT  ABSTRACT  ABSTRACT  ABSTRACT
 -DELETE A        M:  1 dset:   0 I: -999 -999  J: -999 -999  K: -999 -999  L: -999 -999  M: -999 -999  N: -999 -999
 dealloc  dynamic grid ABSTRACT        ABSTRACT  ABSTRACT  ABSTRACT  ABSTRACT  ABSTRACT  ABSTRACT

(The "let a = 0" used to cause corruption errors.  But the M: 2 lines use to be M: 1, and the M: 1 line used to be M: 2.)
I am not sure how to interpret all this diagnostic info.  I just wanted to be sure this is now looking correct.  Or possibly this is a simple example to follow through with the debugger to make sure everything is going down the correct paths.
When I looked at this example when it was giving the error, I got the impression that variable locations were getting messed up.  The opening of coads_climatology was an essential part of causing the error because of different memory managing (compacting?) behaviour for dataset variables and user-defined variables - and it appeared a user-defined variable was getting treated as a dataset variable.

Comment by steven.c.hankin on 7 Dec 2016 21:44 UTC
This diagnostic output doesn't necessarily indicate a problem. The M value is the memory slot occupied by a particular instantiated subset (region) of a variable (a so-called "memory variable" or "mvar" or "mr").  When the script deletes the definition of variable A, it frees all mvars connected to A.  When A gets defined anew, it starts from a blank slate, getting mvars assigned to it, when it needs them.
Mvars are managed by the doubly linked list that is the variables mv_flink and mv_blink ("memory variable forward/back link").  Similarly mblk_flink and mblk_blink manage the "memory blocks".  These are likely candidates as the location where corruption is occurring.
I propose to take the following approach to debugging this -- inviting your comments before I get started:
o The challenge is that the error we are tracking down materializes only after a complex script has been run.  By this point the linked lists (if they are indeed the cause) have become awfully tangled for a human sleuth to follow.  I propose to write a new routine SANITY_CHECK_LINKS that will check the links and output a message if a problem is found.  When running in MODE DIAGNOSTIC, this routine will get called before exiting any of the significant memory management routines -- get_memory, free_memory, etc.  The goal is to see a report of the corruption at the point that it first occurs.  Then using totalview my hope is to be able to more easily track down the cause of the corruption.

Comment by steven.c.hankin on 9 Dec 2016 19:30 UTC
Progress report:
I am now using new routine SANITY_CHECK_MEMORY as a tool (in diagnostic mode, only) to detect when memory corruption occurs.
In the course of early testing I discovered that MR_NOT_IN_USE is being called duplicatively in some cases -- e.g. in IS_RECONCILE and then again in UNPROTECT_CMND_DATA.  In a complex calculation, where the same component data is being reserved for use in two separate upcoming parts of a calculation and memory capacity is hard-pressed, duplicated calls like this could have the effect of placing a component onto the deletion chain (where it becomes vulnerable to automated deletion if more memory is needed) when it is still needed for a future part of the calculation.  If this occurred the data that "mr" pointed to could be over-written by other data.  This matches the symptoms uncovered in Ned's script.

Comment by steven.c.hankin on 3 Jan 2017 17:54 UTC
I am embarrassed to admit that I spent over 6 hours working to track down this bug and still failed.  So it goes ...
I was looking for a corruption of memory -- either the mr table (memory resident variables, a.k.a. "mr's") or the mblk array (the chunk of memory allocated to each memory resident variable).  I wrote a new routine SANITY_CHECK_MEMORY to perform internal checking.  It is now an in-built part of Ferret (invoked whenever CREATE_MEM_VAR is called if the code is compiled debug, and mode diagnostic is ON).   But, bottom line:  no memory corruption of this type seems to be occurring.  I was looking under the wrong rock.  So where else to look?
What I observed is that the error shows up only when the mr table is completely full.  At this point variables get deleted from cache (chosen from the delete priority chain, mr_del_flink) whenever a slot is needed for a new variable to be created.  In Ned's script this situation is not reached until after the first (uncorrupted) plot has been generated.  To see this in totalview put a breakpoint in FIND_MR_SLOT right after "100    IF ( mv_flink( 0 ) . EQ. 0 ) THEN".  This IF tests whether a free mr slot is available.
My belief is that the problem is occurring because the automatic deletion is (incorrectly) deleting an mr that has yet to be used.  For example, given LET v=a+b, Ferret will compute and cache mr's a and b, in order to use them later to compute v.  If for some reason a were (incorrectly) deleted before v had used it, v would be computed using whatever (incorrect) numerical values were found in the new variable occupying a's mr slot.
To successfully debug this problem I believe we need to reproduce it in a simpler example.   The corruption is showing up in the variable PAR_AIR_FILTERED.  The dependency tree of this variable is frightening!
yes? vtree par_air_filtered
   in default dataset Saildrone_sd-128_payloads_20160523_0000UTC_to_20160524_0000UTC_60_s_intervals
   PAR_AIR_FILTERED = PAR_AIR_FILTERED_RESHAPE_FILL[    PAR_AIR_FILTERED_RESHAPE_FILL = IFV ( PAR_AIR_FILTERED_RESHAPE_CD_NR_MASK AND PAR_AIR_FILTERED_RESHAPE_CDS2 LE GAP_
FILL ) THEN PAR_AIR_FILTERED_RESHAPE_NEAR ELSE PAR_AIR_FILTERED_RESHAPE_INTRP
       PAR_AIR_FILTERED_RESHAPE_CD_NR_MASK = IF ( PAR_AIR_FILTERED_RESHAPE_CDS_NRTST EQ -999 ) THEN 1
         PAR_AIR_FILTERED_RESHAPE_CDS_NRTST = MISSING( PAR_AIR_FILTERED_RESHAPE_CDS, -999)
           PAR_AIR_FILTERED_RESHAPE_CDS = PAR_AIR_FILTERED_RESHAPE_CDA + PAR_AIR_FILTERED_RESHAPE_CDB
             PAR_AIR_FILTERED_RESHAPE_CDA = PAR_AIR_FILTERED_RESHAPE[T=@CDA](GT=T_MIN_EQ_AX]
)
               PAR_AIR_FILTERED_RESHAPE = RESHAPE( PAR_AIR_FILTERED_IN, T[)
                 (C001,V***) = RESHAPE( WIND_SPEED_RMS_IN, T[GT=INST_T_AX](GT=INST_T_AX]) )
                   (C029,V***) = T[            PAR_AIR_FILTERED_RESHAPE_CDB = PAR_AIR_FILTERED_RESHAPE[T=@CDB](GT=INST_T_AX]
)
               PAR_AIR_FILTERED_RESHAPE =  (defined above)
       PAR_AIR_FILTERED_RESHAPE_CDS2 = ( PAR_AIR_FILTERED_RESHAPE_CDA_NRTST + PAR_AIR_FILTERED_RESHAPE_CDB_NRTST ) * PAR
_AIR_FILTERED_RESHAPE_CD_NR_MASK
         PAR_AIR_FILTERED_RESHAPE_CDA_NRTST = MISSING( PAR_AIR_FILTERED_RESHAPE_CDA, 0.0)
           PAR_AIR_FILTERED_RESHAPE_CDA =  (defined above)
         PAR_AIR_FILTERED_RESHAPE_CDB_NRTST = MISSING( PAR_AIR_FILTERED_RESHAPE_CDB, 0.0)
           PAR_AIR_FILTERED_RESHAPE_CDB =  (defined above)
         PAR_AIR_FILTERED_RESHAPE_CD_NR_MASK =  (defined above)
       GAP_FILL = 1200
       PAR_AIR_FILTERED_RESHAPE_NEAR = PAR_AIR_FILTERED_RESHAPE_CD_NR_MASK*PAR_AIR_FILTERED_RESHAPE_NR
         PAR_AIR_FILTERED_RESHAPE_CD_NR_MASK =  (defined above)
         PAR_AIR_FILTERED_RESHAPE_NR = PAR_AIR_FILTERED_RESHAPE[          PAR_AIR_FILTERED_RESHAPE =  (defined above)
       PAR_AIR_FILTERED_RESHAPE_INTRP = PAR_AIR_FILTERED_RESHAPE_CDS_MASK*PAR_AIR_FILTERED_RESHAPE_LIN
         PAR_AIR_FILTERED_RESHAPE_CDS_MASK = IF ( PAR_AIR_FILTERED_RESHAPE_CDS LE GAP_FILL ) THEN 1
           PAR_AIR_FILTERED_RESHAPE_CDS =  (defined above)
           GAP_FILL =  (defined above)
         PAR_AIR_FILTERED_RESHAPE_LIN = PAR_AIR_FILTERED_RESHAPE[T=@FLN](T=@FNR]
)
           PAR_AIR_FILTERED_RESHAPE =  (defined above)

Correctly managing the cache dependencies is the responsibility of the Ferret variable mr_protected, which counts the number of descendant variables that depend upon it (or may be a special flag value such as mr_permanent, that indicates other caching behavior).
The problem in the cache logic may be
1. a fundamental flaw that has never been observed, because the automatic cache deletions seldom happen;
or
2. a subtle flaw having to do with the RESHAPE function or the pseudo-variable "T" that is an argument to it.

I created a  simple script that fills the mr table with mr's and then hammers the heck out of the cached variables in new calculations.  It is designed so that a wrong answer is immediately visible.  No matter what complexity I added, though, I could not replicate the problem that Ned's script is triggering.  Here is the test script, should anyone want to add more to it:
! test_auto_cache_delete.jnl  [new files](1=make) [of files](#)

! 1/2017 - try to force Ferret into a complex juggling of dependencies
! and cahce maintenance in order to reproduce the problem seen in trac
! ticket #2485 (from Ned Cokelet)

define symbol nsets ($2"100")
define axis/x=1:100:1 x100

if ($1"0") then
!**************************************
 define axis/x=1:10:1 x10
 define axis/y=1:10:1 y10
 let shape = 0*(X[let one      = 1 + shape
 let ten      = 10 + shape
 let hundred  = 100 + shape
 let thousand = 1000 + shape
 let tenthou  = 10000 + shape
 let hundthou = 100000 + shape
 let million  = 1000000 + shape

 REPEAT/k=1:($nsets)  \
 save/file=test_auto_`k`.nc/clobber one, ten, hundred, thousand, tenthou, hundthou, million

 cancel var/all
!**************************************
endif

REPEAT/k=1:($nsets)  USE test_auto_`k`.nc

let  r_one      = RESHAPE(one,X[gx=x100](gx=x10]+Y[gy=y10])
))
let  r_ten      = RESHAPE(ten,X[ r_hundred  = RESHAPE(hundred,X[gx=x100](gx=x100])
let))
let  r_thousand = RESHAPE(thousand,X[ r_tenthou  = RESHAPE(tenthou,X[gx=x100](gx=x100])
let))
let  r_hundthou = RESHAPE(hundthou,X[ r_million  = RESHAPE(million,X[gx=x100](gx=x100])
let))

let one_2      = r_one
let ten_2      = r_ten
let hundred_2  = r_hundred
let thousand_2 = r_thousand
let tenthou_2  = r_tenthou
let hundthou_2 = r_hundthou
let million_2  = r_million

let one_3      = r_one
let ten_3      = r_ten
let hundred_3  = r_hundred
let thousand_3 = r_thousand
let tenthou_3  = r_tenthou
let hundthou_3 = r_hundthou
let million_3  = r_million

let result_31 = \
one_3 + ten_3 + hundred_3 + thousand_3 + tenthou_3 + hundthou_3 + million_3

let result_21 = \
one_2 + ten_2 + hundred_2 + thousand_2 + tenthou_2 + hundthou_2 + million_2

let result_32 = \
one_3 + ten_3 + hundred_3 + thousand_3 + tenthou_3 + hundthou_3 + million_3

let result_22 = \
one_2 + ten_2 + hundred_2 + thousand_2 + tenthou_2 + hundthou_2 + million_2

let result = result_31 + result_21 + result_32 + result_22 + \
r_one + r_ten + r_hundred + r_thousand + r_tenthou + r_hundthou + r_million

sp rm -f test_auto.out
SET MODE DIAG
REPEAT/k=1:($nsets)  LOAD result[ list/nohead/i=50/file=test_auto.out/append result[d=test_auto_`k`.nc](d=test_auto_`k`.nc]
REPEAT/k=1:($nsets))

vtree result

Attachment from ansley.b.manke on 16 Nov 2016 01:41 UTC
ensuring we have the example scripts that fail
REPLACE THIS TEXT WITH UPLOADED FILE ./attachments/TRAC_2485_GIT_1757/ned2.tar.gz

In investigating #1886, the function HASH_NAME(name, npiles) is called.  (There is also a TM_HASH_NAME that is basically the same.)  At some point the value returned was used as an index into an array, and if there is a valid value at that location, the variables are assumed to be the same.  This is fine if the array is large relative to the number of variables as there is very small chance of collision.  But with supporting more variables, if the array size is not increased, the chance of collision increases.  I am suspicious that a collision in this is the source of using the incorrect variable.

repo: NOAA-PMEL/Ferret

license: Unlicense


text:
Plot of time axis causes message "STOP,2" and Ferret quits

Reported by Ansley B Manke on 7 May 2004 22:08 UTC
(Wittenberg, 7-May-2004)
yes? use n99_100.nc
*** NOTE: Units on axis "SIGMA_TS" are not recognized: sigma_level
*** NOTE: They will not be convertible:
yes? plot/step=con n       ! this works
yes? plot/ov n             ! this too
yes? plot n                ! but this crashes Ferret!
STOP ,2 statement executed
Looks like the data is OK, but the plot routine has a bug, possibly in the
part that sets up the time axis limits...
Andrew
Here's more about the file:
yes? use n99_100.nc
yes? sh grid n
GRID GST1
name       axis              # pts   start                end
normal    X
normal    Y
SIGMA_TS  Z (sigma_level       1 r-  0.99665              0.99665
TIME      TIME                 2 r   07-FEB-0599 12:00    07-FEB-0600 12:00
yes? sh axis time
name       axis              # pts   start                end
TIME      TIME                 2 r   07-FEB-0599 12:00    07-FEB-0600 12:00
T0 = 01-JAN-0001 00:00:00
Axis span (to cell edges) = 730
yes? list t[tboxgt=time
TIME: 09-AUG-0598 00:00 to 09-AUG-0600 00:00
Column  1: T is T (axis TIME)
Column  2: TBOX is TBOX (axis TIME)
T   TBOX
FEB-0599 / 1:  218452.  365.0
FEB-0600 / 2:  218818.  365.0
This tries to plot the time axis with TIME3,
ITIME1 = 99020100000005
ITIME2 = 00030200000006
The routine expects that elements 5:10 of these WHOI
dates are "010000"  and ITIME2 is not.
SETAX has called FIXTIM, with itype=3, returns
00030200000006 rather than 00030100000006.
Turns out that numdm1 when it is seeing if we are in a
leap year, does not take the CENTURY into account when
seeing what year we are in and whether it is a leap year!
Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/882

Comment by Ansley.B.Manke@noaa.gov on 7 May 2004 22:18 UTC
Here is an example that does not depend on a data set.
Time axis ends in February of the first year of a century,
which is not a leap year.  The adjustment of the start
date for plotting the time axis tries to form a date with
day 1 of a month, but misses because of this error
yes? def
axis/t="07-FEB-1899:12:00":"07-FEB-1900:12:00":365.2425/units=days/t0="1-jan-1800"
tax
yes? let v = tsequence({55,65})
yes? plot v[gt=tax@asn]
STOP: ,2

Comment by Ansley.B.Manke@noaa.gov on 8 Jun 2004 21:26 UTC
ppl routine numdn1.F did not take century into account
when checking for leap years

Comment by ansley.b.manke on 4 Nov 2008 19:49 UTC
Adding Andrew to the CC list where he is mentioned in the ticket comments.

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
Remove total PTC output from all models

Fixes #644
Remove cf_ptc_total from cashloan and common.
Ignoring legacy cmod_ippppa.

Issue to move old IPP financial model code to attic: https://github.com/NREL/ssc/issues/1074

source: gharchive/pull-request

repo: NREL/ssc


text:
feat: Add strict mode for marshal, enforce use of uri and default to about:blank for type.

closes #11

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (eef42ea) to head (ed4b651).

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #12   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            1         1
  Lines           70        76    +6
  Branches         6         9    +3
=========================================
+ Hits            70        76    +6

Flag
Coverage Δ

unittests
100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: EdgyEdgemond codecov-commenter


text:
TypeScript response code handling bug

Generating typescript client code for the controller below, resulted in exception throw statements for status codes that are not errors.
public class FooController : ApiController
    {
        [Route("foos/")]
        [SwaggerResponse(200, null)]
        [SwaggerResponse(201, null)]
        [SwaggerResponse(400, null)]
        [SwaggerResponse(500, null)]
        public Foo[] GetFoos([FromUri] Bar[] bars)
        {
            return new Foo[0];
        }
    }

Before:
protected processGetFoos(response: Response): void {
    const responseText = response.text();
    const status = response.status;

    if (status === 200) {
        return null;
    } else if (status === 201) {
        this.throwException("A server error occurred.", status, responseText);
    } else if (status === 400) {
        this.throwException("A server error occurred.", status, responseText);
    } else if (status === 500) {
        this.throwException("A server error occurred.", status, responseText);
    } else if (status !== 200 && status !== 204) {
        this.throwException("An unexpected server error occurred.", status, responseText);
    }
    return null;
}

Clearly, a HTTP 201 is not supposed to be handled as an error.
After:
protected processGetFoos(response: Response): void {
    const responseText = response.text();
    const status = response.status;

    if (status === 200) {
        return null;
    } else if (status === 201) {
        return null;
    } else if (status === 400) {
        this.throwException("A server error occurred.", status, responseText);
    } else if (status === 500) {
        this.throwException("A server error occurred.", status, responseText);
    } else if (status !== 200 && status !== 204) {
        this.throwException("An unexpected server error occurred.", status, responseText);
    }
    return null;
}

The same bug is fixed in CSharpOperationModel as well, though I didn't include a test for that.

This is not a bug but by design - what if 200 and 201 do not return the same response type? Then you would have not compilable code... Returning from 200 and 201 is only allowed if both have the same Response type...

I get it.
I actually wasn't returing both 200 and 201 in my case but due to the fact that I used the [ResponseType(typeof(Model))] attribute as well (that's the one from the System.Web.Http.Description namespace), a 200 was added in addition to my 201.
So it turns out that this:
    public class FooController : ApiController
    {
        [Route("foos/")]
        [System.Web.Http.Description.ResponseType(typeof(Foo[]))]
        [SwaggerResponse(201, null)]
        public IHttpActionResult GetFoos([FromUri] Bar[] bars)
        {
            return Ok(new Foo[0]);
        }
    }

leads to:
protected processGetFoos(response: Response): Foo[] {
    const responseText = response.text();
    const status = response.status;

    if (status === 200) {
        let result200: Foo[] = null;
        let resultData200 = responseText === "" ? null : JSON.parse(responseText, this.jsonParseReviver);
        if (resultData200 && resultData200.constructor === Array) {
            result200 = [];
            for (let item of resultData200)
                result200.push(Foo.fromJS(item));
        }
        return result200;
    } else if (status === 201) {
        this.throwException("A server error occurred.", status, responseText);
    } else if (status !== 200 && status !== 204) {
        this.throwException("An unexpected server error occurred.", status, responseText);
    }
    return null;
}

I guess I'll be removing all [ResponseType(typeof(...))] declarations in my code base where ever I have a [SwaggerResponse(...)] as well.
As a final thought, wouldn't it be great if I had been warned by the client generator that multiple non-error responses were declared? The client code generated in that case just doesn't make much sense anyhow. I'd say: either allow it and generate sensible code or prohibit it.

There was some discussion in #622 related to handling multiple non-exceptional response types

@sandord How did you end up handling this?

Since I wasn't really returning multiple success codes after all (a 200 snuck in due to the ResponseModelAttribute, next to my 201), I didn't have to.

authors: grovesNL rsuter sandord

repo: NSwag/NSwag

license: mit

license_source: bigquery


text:
Change the default text and details link for path progress assignments

The default text for path progress assignments should be "Complete activities on the path {path name}."
The path name can be provided as the hyperlink to the path rather than providing a hyperlinked "details".

Path progress's link to the path is now in the "Go to Path"

The default text is now fixed with #505

authors: FeynmanDNA scboesch

repo: NUS-ALSET/achievements

license: MIT


text:
Adds receiving ActuatorRequests and sending SensorMeasurements

Mostly taken from cyberbotics/webots#2888. Removed RoboCup-specific stuff. Changed some of the code.
It would be great if someone could look at the code because I'm sure it can be made better. And I've never properly used smart pointers before so I wouldn't be surprised if I've messed things up there.
I don't think image compression is finished so I've commented it out.

@TrentHouliston do you think we should be using weak pointers instead of raw pointers for the Webots objects which we don't have ownership of?

@TrentHouliston do you think we should be using weak pointers instead of raw pointers for the Webots objects which we don't have ownership of?

Do we know that the objects are managed by shared_ptrs?

std::weak_ptr is a smart pointer that holds a non-owning ("weak") reference to an object that is managed by std::shared_ptr.

Do we know that the objects are managed by shared_ptrs?

Ahh, I misunderstood that... I think much of webots is written in C, so that seems unlikely

I think that this PR should be merged, and that we can improve what exists such that we can run keyboardwalk in follow up PRs.

I reckon it'd be pretty cool if we could get this merged :wink:
@Bidski @CMurtagh-LGTM

authors: Bidski KipHamiltons ysims

repo: NUbots/NUWebots


text:
Code fails to run in both jupyter and console

Sorry for the less than descriptive title, but I wasn't sure how else to title it.
I've got a 4gb Jetson Nano (SeeedStudio Jetson Recomputer J2010 carrier board) with a 128gb SSD as the root storage device. It's running JetPack 4.6 (output of 'apt-cache show nvidia-jetpack' below)
`$ sudo apt-cache show nvidia-jetpack
Package: nvidia-jetpack
Version: 4.6-b199
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6-b199), nvidia-opencv (= 4.6-b199), nvidia-cudnn8 (= 4.6-b199), nvidia-tensorrt (= 4.6-b199), nvidia-visionworks (= 4.6-b199), nvidia-container (= 4.6-b199), nvidia-vpi (= 4.6-b199), nvidia-l4t-jetson-multimedia-api (>> 32.6-0), nvidia-l4t-jetson-multimedia-api (<< 32.7-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6-b199_arm64.deb
Size: 29368
SHA256: 69df11e22e2c8406fe281fe6fc27c7d40a13ed668e508a592a6785d40ea71669
SHA1: 5c678b8762acc54f85b4334f92d9bb084858907a
MD5sum: 1b96cd72f2a434e887f98912061d8cfb
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8
Package: nvidia-jetpack
Version: 4.6-b197
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6-b197), nvidia-opencv (= 4.6-b197), nvidia-cudnn8 (= 4.6-b197), nvidia-tensorrt (= 4.6-b197), nvidia-visionworks (= 4.6-b197), nvidia-container (= 4.6-b197), nvidia-vpi (= 4.6-b197), nvidia-l4t-jetson-multimedia-api (>> 32.6-0), nvidia-l4t-jetson-multimedia-api (<< 32.7-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6-b197_arm64.deb
Size: 29356
SHA256: 104cd0c1efefe5865753ec9b0b148a534ffdcc9bae525637c7532b309ed44aa0
SHA1: 8cca8b9ebb21feafbbd20c2984bd9b329a202624
MD5sum: 463d4303429f163b97207827965e8fe0
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8
`
I've set up a python 3.6 virtualenv and followed the installation instructions for all required packages. There were no errors during any of the installations and I've verified all of the packages import properly from the python command line. When I run the jupyter notebook 'live_demo.ipynb' I am able to run all of the steps up until the following step:
`import torch2trt
model_trt = torch2trt.torch2trt(model, [data], fp16_mode=True, max_workspace_size=1<<25)`
When I attempt to run that step the systems thinks about it for a bit and then a dialog pops up that says the python kernel has crashed and will be automatically restarted. I cannot get past this step.
To help debug/diagnose I took all of the code from the notebook and incrementally added it to a python file to see if I could reproduce the issue. The code I have so far is:
`import cv2
import json
import trt_pose.coco
import trt_pose.models
import torch
import torch2trt
from torch2trt import TRTModule
import time
import torchvision.transforms as transforms
import PIL.Image
from trt_pose.draw_objects import DrawObjects
from trt_pose.parse_objects import ParseObjects
from jetcam.usb_camera import USBCamera
from jetcam.csi_camera import CSICamera
from jetcam.utils import bgr8_to_jpeg
import ipywidgets
from IPython.display import display
with open('human_pose.json', 'r') as f:
human_pose = json.load(f)
topology = trt_pose.coco.coco_category_to_topology(human_pose)
num_parts = len(human_pose['keypoints'])
num_links = len(human_pose['skeleton'])
model = trt_pose.models.resnet18_baseline_att(num_parts, 2 * num_links).cuda().eval()
MODEL_WEIGHTS = 'resnet18_baseline_att_224x224_A_epoch_249.pth'
model.load_state_dict(torch.load(MODEL_WEIGHTS))
WIDTH = 224
HEIGHT = 224
data = torch.zeros((1, 3, HEIGHT, WIDTH)).cuda()
print("Calling torch2trt.torch2trt\n")
model_trt = torch2trt.torch2trt(model, [data], fp16_mode=True, max_workspace_size=1<<25)
print("Done\n")
OPTIMIZED_MODEL = 'resnet18_baseline_att_224x224_A_epoch_249_trt.pth'
print("Calling torch.save\n")
torch.save(model_trt.state_dict(), OPTIMIZED_MODEL)
print("Done\n")`
When I run this code I see "Calling torch2trt.torch2trt" in the console and then, after a pause, I see the following error repeated many times on the console:
[TensorRT] ERROR: 3: [builderConfig.cpp::canRunOnDLA::341] Error Code 3: Internal Error (Parameter check failed at: optimizer/api/builderConfig.cpp::canRunOnDLA::341, condition: dlaEngineCount > 0 )
After that the system will seem to hang. The desktop will display a low memory warning (If I run 'watch -n1 free -h' in another console window I can see free memory drop from 3+GB to as little as 96MB). After some time the process just reports "Killed" and exits back to the command line.
I am at a loss. Can you please provide any helpful information you have that might help me correct this issue and continue?
Thanks
Marc

bump

@marcjasner have you figured out what the problem was? However, in my case I am able to pass this point, but I also get the same message written in the console. Also, the FPS is lower (8) than claimed (22)? My program then crashes at line: 	torch.save(model_trt.state_dict(), OPTIMIZED_MODEL), where the optimized model weights should be saved..

@janezlapajne, no I never resolved that issue. I ended up investigating other pose detection methods and found the jetson_inference posenet code worked reliably and gave pretty good performance (about 17FPS)

Thanks for telling me, I wish I would find out sooner - wouldn't spend that much time on it.. can you redirect me to the repo/model? I hope it can be quickly set and tested? Probably is this one (?): https://github.com/dusty-nv/jetson-inference/blob/master/docs/posenet.md
If you have any other suggestions etc. please let me know. I would just like to make it work ASAP.

@janezlapajne yes, that is the one

Yep, as @dusty-nv pointed out, that is the correct one. Also, Dusty is super helpful on the Nvidia forums. He's helped me a number of times!
I think you'll find getting the code compiled and getting the posenet sample running is pretty straightforward and easy, and adapting it to any projects you're working on should be similarly easy.

Hello, thank you both! Yes, I agree, @dusty-nv dusty never disappoints💪
Also, this jetson-inference package is amazing - made the pose model work in literally a couple of minutes, inside a docker container. Sincerely, dusty, great work! I really appreciate it! A few months ago, I also used the SSD detector from jetson-inference and it worked like a charm. The retraining process has worked out of the box with an additional script for the automatic download procedure of the open-images dataset. Really helpful if you want to quickly test and prototype something.
Anyways, to conclude I will ask something else (correct me if this is not the right place to ask). We plan to use two models concurrently on a Jetson nano - a pose model and detection model i.e. preferably yolov7 with via Deepstream package. Can Jetson Nano handle resources appropriately in such cases? Thank you!

Hello, thank you both! Yes, I agree, @dusty-nv dusty never disappoints💪
Also, this jetson-inference package is amazing - made the pose model work in literally a couple of minutes, inside a docker container. Sincerely, dusty, great work! I really appreciate it! A few months ago I also used the SSD detector from jetson-inference and it worked like a charm. The retraining process have worked out of the box with additional automatic download procedure of open-images. Really helpful if you want to quickly test and prototype something.

a pose model and detection model i.e. preferably yolov7 with via Deepstream package. Can Jetson Nano handle resources appropriately in such cases?

Due to it's limited compute resources, it's hard to say with Nano, so you may need to play around with it - also, deepstream supports pose estimation (https://docs.nvidia.com/tao/tao-toolkit/text/bodypose_estimation/bodyposenet.html), so ultimately you may find that doing both detection+pose in deepstream gives you better performance.

@dusty-nv ok, will see how it goes. Will report to the forum if I have any further questions.

@marcjasner When I run the jupyter notebook 'live_demo.ipynb' I am able to run all of the steps up until the following step:
`import torch2trt
model_trt = torch2trt.torch2trt(model, [data], fp16_mode=True, max_workspace_size=1<<25)`
I have a similar issue, just that it doesn't crash , the code just stays stuck... Does anyone have some advice for me?

authors: ArcadeHustle dusty-nv janezlapajne marcjasner patriciamold33


text:
Getting capped at 90 samples per label when creating dataset

Hi,
I am making a dataset where I have hundreds of images per label. However, when I create a dataset with this data in DIGITS it only includes 90 of them maximum per label. I have attached an image of what I mean here. I tried using the mnist data and did not experience this cap using the exact same settings. Originally I thought it was because my images are each around 30kb each and maybe Digits cannot handle large sizes, but I also tried reducing the jpeg quality of the images very low so they were each 3-4kb each (like the mnist example) and instead was capped at 100. Any help would be greatly appreciated!
Note: I am using the master versions of DIGITS and using the Use S3 tab

Update: Looking through the digits/tools/create_db.py I think the problem could be in this function which specifies 100 as the image count
def _calculate_batch_size(image_count, is_hdf5=False, hdf5_dset_limit=None,                           image_channels=None, image_height=None, image_width=None):     """     Calculates an appropriate batch size for creating this database     """     if is_hdf5 and hdf5_dset_limit is not None:         return min(100, image_count, hdf5_dset_limit / (image_channels * image_height * image_width))     else:         return min(100, image_count)

Yeah I have encountered this as well.  Looks like the cap is specified here
https://github.com/NVIDIA/DIGITS/blob/master/digits/tools/parse_s3.py#L372

repo: NVIDIA/DIGITS


text:
Problem installing caffe for digits

Hi, Im trying to install nvidia's version of caffe, with steps provided here, and everything works, until I run: make runtest, it outputs this:
.build_release/tools/caffe
dyld: Library not loaded: libcaffe-nv.so.0
Referenced from: /Users/caffeteria/caffe/.build_release/tools/caffe
Reason: image not found
make: *** [runtest] Trace/BPT trap: 5
can anyone help how to solve this?
OS 10.10.2, I mostly used brew and pip

vanilla version of caffe compiled just fine

Having trouble with the make all command. Any ideas?
~/caffe$ make all
LD -o .build_release/lib/libcaffe-nv.so.0.10.0
/usr/bin/ld: cannot find -lcudnn
collect2: error: ld returned 1 exit status
make: *** [.build_release/lib/libcaffe-nv.so.0.10.0] Error 1

@Tutufa, Mac is not technically a supported OS, as seen here. However, Rudolph seems to have had some luck with it (see #32 and #33). In your case, it looks like the shared objects aren't being created correctly, as the libraries should end with .dylib on a Mac instead of .so. @RudolphV, did you run into this issue?
@cicero19, your problem is unrelated. It seems like you need to either install cuDNN or turn off the USE_CUDNN flag in your Makefile.config.

The only two issues I ran into was #32 and #33 and the workarounds discussed in those threads fixed the problems.
Caffe (ver 0.10.0) installed and tested fine with cuDNN. I followed the installation instructions from here.
I'm running the developer seed of Mac OS X 10.10.3 and the latest corresponding drivers from NVIDIA (NVIDIA Web Driver: 343.02.03b04, CUDA 7.0.35 (CUDA Driver Version: 7.0.29, GPU Driver Version 10.3.5 (343.02.03b4)).

@lukeyeager thanks, I figured out that issue. I was able to start digits but it was unable to find my GPU:
Something went wrong when loading libcudart.so
No GPUs found. Assuming CPU-only mode.
How do I fix this libcudart.so error?
When I do make runtest on caffe it tells me: CUDA driver version is insufficient for CUDA runtime version. In my usr/local folder I have one folder named cuda and another named cuda-7.0. Not sure which one it is using.

CUDA driver version is insufficient for CUDA runtime version

It sounds like you need to update your driver.
If that doesn't work for you, make sure you have the latest changes for DIGITS that include the bugfix for #33.

I updated the driver with
$ sudo apt-get install nvidia-current
Now Digits doesn't start. I checked the file from #33 but I am not on a Mac so not sure how it is applicable.

|   ()/ __()_   / |
| |) | | ( | | | | _
|/||_|| || |__/
OSError: libcudart.so: cannot open shared object file: No such file or directory
Try setting your LD_LIBRARY_PATH
Traceback (most recent call last):
File "digits-devserver", line 40, in
from digits.webapp import app, socketio, scheduler
File "/home/ciceromar/digits/digits/webapp.py", line 11, in
import digits.scheduler
File "/home/ciceromar/digits/digits/scheduler.py", line 18, in
from model import ModelJob, tasks as model_tasks
File "/home/ciceromar/digits/digits/model/init.py", line 3, in
from job import ModelJob
File "/home/ciceromar/digits/digits/model/job.py", line 4, in
from . import tasks
File "/home/ciceromar/digits/digits/model/tasks/init.py", line 4, in
from caffe_train import CaffeTrainTask
File "/home/ciceromar/digits/digits/model/tasks/caffe_train.py", line 5, in
import caffe
File "/home/ciceromar/caffe/python/caffe/init.py", line 1, in
from .pycaffe import Net, SGDSolver
File "/home/ciceromar/caffe/python/caffe/pycaffe.py", line 13, in
from ._caffe import Net, SGDSolver
ImportError: libcudart.so.7.0: cannot open shared object file: No such file or directory

I am not on a Mac so not sure how it is applicable

Oh sorry. You really should have put this in a separate issue, I got your issue mixed up with Tutufa's.
As it says in the output you pasted above:

Try setting your LD_LIBRARY_PATH

You're looking for something like this:
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH`

It sounds like @cicero19 solved his problem. @Tutufa, were you able to get DIGITS running?

@lukeyeager No, I use vanilla caffe currently, runs fine. But nvidia brunch gives same mistake, when I try to do make runtest:
Kovalenkos-MacBook-Pro:caffe_nvidia kovalenko$ make runtest
.build_release/tools/caffe
dyld: Library not loaded: libcaffe-nv.so.0
Referenced from: /Users/kovalenko/caffe_nvidia/.build_release/tools/caffe
Reason: image not found
make: *** [runtest] Trace/BPT trap: 5
Quite sad, digits looks very sexy on webinar I watched few days ago.

Quite sad

Indeed. @moconnor725, can you help figure out how to fix this on OS X? A hacky solution is fine for now until we fix up the cmake build.

Yes, I was able to resolve my issue with a graphics driver update and updated CUDA driver. Thanks for your fast and helpful support. Keep up the good work!

I migrated to AWS + ubuntu, digits runs fine so far. Thanks to all, hope os x version will arrive soon (-:

I finished all the install steps. However, the results is totally failed.

|   ()/ __()_   / |
| |) | | ( | | | | _
|/||_|| || |__/
Welcome to the DiGiTS config module.
Where is caffe installed? (enter "SYS" if installed system-wide)
[default is /home/yingyu/Downloads/digits-1.0/caffe]
(q to quit) >>>
Where would you like to store jobs?
[default is /home/yingyu/.digits/jobs]
(q to quit) >>>
What is the minimum log level that you want to save to your logfile? [error/warning/info/debug]
[default is info]
(q to quit) >>>
New config:
gpu_list -
secret_key - 27cbcd5f161ce2d2067db358
log_level - info
jobs_dir - /home/yingyu/.digits/jobs
caffe_root - /home/yingyu/Downloads/digits-1.0/caffe
Traceback (most recent call last):
File "digits/digits-devserver", line 40, in
from digits.webapp import app, socketio, scheduler
File "/home/yingyu/Downloads/digits-1.0/digits/digits/webapp.py", line 11, in
import digits.scheduler
File "/home/yingyu/Downloads/digits-1.0/digits/digits/scheduler.py", line 18, in
from model import ModelJob, tasks as model_tasks
File "/home/yingyu/Downloads/digits-1.0/digits/digits/model/init.py", line 3, in
from job import ModelJob
File "/home/yingyu/Downloads/digits-1.0/digits/digits/model/job.py", line 4, in
from . import tasks
File "/home/yingyu/Downloads/digits-1.0/digits/digits/model/tasks/init.py", line 4, in
from caffe_train import CaffeTrainTask
File "/home/yingyu/Downloads/digits-1.0/digits/digits/model/tasks/caffe_train.py", line 5, in
import caffe
File "/home/yingyu/Downloads/digits-1.0/caffe/python/caffe/init.py", line 1, in
from .pycaffe import Net, SGDSolver
File "/home/yingyu/Downloads/digits-1.0/caffe/python/caffe/pycaffe.py", line 13, in
from ._caffe import Net, SGDSolver
ImportError: libcudart.so.7.0: cannot open shared object file: No such file or directory

libcudart.so.7.0: cannot open shared object file: No such file or directory

See https://github.com/NVIDIA/DIGITS/issues/54#issuecomment-91031586 and https://github.com/NVIDIA/DIGITS/issues/8#issuecomment-84061528.
I've updated the install instructions now to include this step.

Hi, Im tryning to compile Nvidia-Caffe to run DIGITS on Mac (OSX 10.10.3)
I'm experiencing the same issue as @Tutufa :

Seans-MacBook-Pro:caffe sean$ make runtest

.build_release/tools/caffe
dyld: Library not loaded: libcaffe-nv.so.0
Referenced from: /Users/sean/caffe/.build_release/tools/caffe
Reason: image not found
make: *** [runtest] Trace/BPT trap: 5

Is there any know solutions yet?
P.S. I'm not familiar with Caffe or DIGITS, plain noob

Can you try the CMake build instead of the Make build? I've been meaning to get around to investigating this issue, but I've been busy with other things.

@lukeyeager I tried with CMake - worked! thanks.
but something weird, I wanted to test that Caffe works so I ran the mnist example,
but 'build' alias to '.build_release' and '.build_release' were not exist.
I simply copied the folder from 'vanilla' installation of Caffe,
Is it a know issue? did I skip something?

@lukeyeager okay, I got another problem:
I tried to 'make pycaffe' and got the following error:

-- Boost version: 1.57.0
-- Found the following Boost libraries:
--   system
--   thread
-- Found glog    (include: /usr/local/include, library: /usr/local/lib/libglog.dylib)
-- Found gflags  (include: /usr/local/include, library: /usr/local/lib/libgflags.dylib)
-- Found PROTOBUF Compiler: /usr/local/bin/protoc
-- Found lmdb    (include: /usr/local/include, library: /usr/local/lib/liblmdb.dylib)
-- Found LevelDB (include: /usr/local/include, library: /usr/local/lib/libleveldb.dylib)
-- Found Snappy  (include: /usr/local/include, library: /usr/local/lib/libsnappy.dylib)
-- CUDA detected: 7.0
-- Found cuDNN (include: /Users/sean/Documents/cuDNN, library: /Users/sean/Documents/cuDNN/libcudnn.6.5.dylib)
-- Added CUDA NVCC flags for: sm_30
-- OpenCV found (/usr/local/share/OpenCV)
-- Found vecLib as part of Accelerate.framework
-- NumPy ver. 1.9.2 found (include: /usr/local/lib/python2.7/site-packages/numpy/core/include)
-- Boost version: 1.57.0
-- Found the following Boost libraries:
-- python
-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE)
--
-- ******************* Caffe Configuration Summary *******************
-- General:
--   Version           :    (Caffe doesn't declare its version in headers)
--   Git               :   v0.12.1-dirty
--   System            :   Darwin
--   C++ compiler      :   /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
--   Release CXX flags :   -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Debug CXX flags   :   -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Build type        :   Release
--
--   BUILD_SHARED_LIBS :   ON
--   BUILD_python      :   ON
--   BUILD_matlab      :   OFF
--   BUILD_docs        :   ON
--   CPU_ONLY          :   OFF
--
-- Dependencies:
--   BLAS              :   Yes (vecLib)
--   Boost             :   Yes (ver. 1.57)
--   glog              :   Yes
--   gflags            :   Yes
--   protobuf          :   Yes (ver. 2.6.1)
--   lmdb              :   Yes (ver. 0.9.14)
--   Snappy            :   Yes (ver. 1.1.3)
--   LevelDB           :   Yes (ver. 1.18)
--   OpenCV            :   Yes (ver. 3.0.0)
--   CUDA              :   Yes (ver. 7.0)
--
-- NVIDIA CUDA:
--   Target GPU(s)     :   Auto
--   GPU arch(s)       :   sm_30
--   cuDNN             :   Yes
--
-- Python:
--   Interpreter       :   /usr/local/Cellar/python/2.7.10_2/bin/python (ver. 2.7.10)
--   Libraries         :   /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib (ver 2.7.10)
--   NumPy             :   /usr/local/lib/python2.7/site-packages/numpy/core/include (ver 1.9.2)
--
-- Documentaion:
--   Doxygen           :   No
--   config_file       :
--
-- Install:
--   Install path      :   /Users/sean/caffe/install
--
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/sean/caffe
[  1%] Built target proto
Linking CXX shared library ../../lib/libcaffe-nv.dylib
Undefined symbols for architecture x86_64:
"caffe::SolverState::SolverState()", referenced from:
caffe::Solver::Restore(char const*) in solver.cpp.o
caffe::Solver::Snapshot() in solver.cpp.o
caffe::Solver::Restore(char const*) in solver.cpp.o
caffe::Solver::Snapshot() in solver.cpp.o
"caffe::SolverState::~SolverState()", referenced from:
caffe::Solver::Restore(char const*) in solver.cpp.o
caffe::Solver::Snapshot() in solver.cpp.o
caffe::Solver::Restore(char const*) in solver.cpp.o
caffe::Solver::Snapshot() in solver.cpp.o
[...]
"caffe::ParamSpec::~ParamSpec()", referenced from:
caffe::Net::Init(caffe::NetParameter const&) in net.cpp.o
caffe::Net::GetLearningRateAndWeightDecay() in net.cpp.o
caffe::Net::Init(caffe::NetParameter const&) in net.cpp.o
caffe::Net::GetLearningRateAndWeightDecay() in net.cpp.o
"typeinfo for caffe::LayerParameter", referenced from:
___cxx_global_var_init30 in layer_factory.cpp.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [lib/libcaffe-nv.0.12.1.dylib] Error 1
make[2]: *** [src/caffe/CMakeFiles/caffe.dir/all] Error 2
make[1]: *** [python/CMakeFiles/pycaffe.dir/rule] Error 2
make: *** [python/CMakeFiles/pycaffe.dir/rule] Error 2

I wanted to test that Caffe works so I ran the mnist example, but 'build' alias to '.build_release' and '.build_release' were not exist.

The correct way to use CMake is:
mkdir build && cd build
cmake ..
make
make py

@lukeyeager Thanks, I'm using the GUI version of CMake, so I guessed that 'cmake ..' is replaced by clicking 'Generate' while 'Where to build binaries' is pointing to 'caffe/build', am I right?
If so, 'make py' print the following error:

make: *** No rule to make target `py'.  Stop.

the parameters I used in CMake:

Boost version: 1.57.0
Found the following Boost libraries:
system
thread
Found glog    (include: /usr/local/include, library: /usr/local/lib/libglog.dylib)
Found gflags  (include: /usr/local/include, library: /usr/local/lib/libgflags.dylib)
Found PROTOBUF Compiler: /usr/local/bin/protoc
Found lmdb    (include: /usr/local/include, library: /usr/local/lib/liblmdb.dylib)
Found LevelDB (include: /usr/local/include, library: /usr/local/lib/libleveldb.dylib)
Found Snappy  (include: /usr/local/include, library: /usr/local/lib/libsnappy.dylib)
CUDA detected: 7.0
Found cuDNN (include: /Users/sean/Documents/cuDNN, library: /Users/sean/Documents/cuDNN/libcudnn.6.5.dylib)
Added CUDA NVCC flags for: sm_30
OpenCV found (/usr/local/share/OpenCV)
Found vecLib as part of Accelerate.framework
NumPy ver. 1.9.2 found (include: /usr/local/lib/python2.7/site-packages/numpy/core/include)
Boost version: 1.57.0
Found the following Boost libraries:
python
Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE)

******************* Caffe Configuration Summary *******************
General:
Version           :    (Caffe doesn't declare its version in headers)
Git               :   v0.12.1-dirty
System            :   Darwin
C++ compiler      :   /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
Release CXX flags :   -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
Debug CXX flags   :   -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
Build type        :   Release

BUILD_SHARED_LIBS :   ON
BUILD_python      :   ON
BUILD_matlab      :   OFF
BUILD_docs        :   ON
CPU_ONLY          :   OFF

Dependencies:
BLAS              :   Yes (vecLib)
Boost             :   Yes (ver. 1.57)
glog              :   Yes
gflags            :   Yes
protobuf          :   Yes (ver. 2.6.1)
lmdb              :   Yes (ver. 0.9.14)
Snappy            :   Yes (ver. 1.1.3)
LevelDB           :   Yes (ver. 1.18)
OpenCV            :   Yes (ver. 3.0.0)
CUDA              :   Yes (ver. 7.0)

NVIDIA CUDA:
Target GPU(s)     :   Auto
GPU arch(s)       :   sm_30
cuDNN             :   Yes

Python:
Interpreter       :   /usr/local/Cellar/python/2.7.10_2/bin/python (ver. 2.7.10)
Libraries         :   /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib (ver 2.7.10)
NumPy             :   /usr/local/lib/python2.7/site-packages/numpy/core/include (ver 1.9.2)

Documentaion:
Doxygen           :   No
config_file       :

Install:
Install path      :   /Users/sean/caffe/build/install

I'm using the GUI version of CMake, so I guessed that 'cmake ..' is replaced by clicking 'Generate' while 'Where to build binaries' is pointing to 'caffe/build', am I right?

Sounds right to me.

If so, 'make py' print the following error:

Whoops, nevermind on the "make py". I've updated my instructions above. For me, when I run make, it builds pycaffe for me automatically:
Linking CXX executable caffe-nv
[100%] Built target extract_features
Linking CXX executable caffe
[100%] Built target caffe.bin
Linking CXX shared library ../lib/caffe-nv.so
Creating symlink /home/lyeager/caffe/python/caffe/_caffe.so -> /home/lyeager/caffe/build/lib/_caffe.so
[100%] Built target pycaffe

Does it not build for you? Check and see if python/caffe/_caffe.so exists.

@lukeyeager it does, <python/caffe/_caffe.so> exist and I can < import caffe > through ipython (if PYTHONPATH set correctly). so it seems it all good, I can even fire up DIGITS with < ./digits-devserver --config > ! thanks a lot!

authors: BINARHO RudolphV Tutufa cicero19 lukeyeager qwersdf


text:
Torch - report error on job summary

Introduce a wrapper script to catch and report errors
in a more straightforward way. The main error is now
shown on the job summary window, in big, red letters.

The error now appears on the job summary. For example, on an attempt to train a model using incompatible input scale:

This seems to be working for me. It caught cuDNN and size mismatch errors well.
I'd like to merge this before RC2 - too risky?

Thanks Luke. I think it's not too risky.

authors: gheinrich lukeyeager


text:
Adds a build arg to allow overriding TE when building rosetta images

This code path is not used in the nightly/presubmit CI and only used internally for testing new changes.
The instructions were updated to demonstrate how to use this, but here are example build invocations to validate:
ROSETTA_BASE=pax

# vanilla
docker buildx build --build-context jax-toolbox=. --tag rosetta-${ROSETTA_BASE}:latest -f rosetta/Dockerfile.${ROSETTA_BASE} --build-arg BASE_IMAGE=ghcr.io/nvidia/upstream-${ROSETTA_BASE}:mealkit-2024-01-13 .

# branch
docker buildx build --build-arg UPDATED_TE_REF=release_v1.3 --build-context jax-toolbox=. --tag rosetta-pax:test-te-branch -f rosetta/Dockerfile.${ROSETTA_BASE} --build-arg BASE_IMAGE=ghcr.io/nvidia/upstream-${ROSETTA_BASE}:mealkit-2024-01-13 .

# pull-request
docker buildx build --build-arg UPDATED_TE_REF=pull/609/head --build-context jax-toolbox=. --tag rosetta-pax:test-te-pr -f rosetta/Dockerfile.${ROSETTA_BASE} --build-arg BASE_IMAGE=ghcr.io/nvidia/upstream-${ROSETTA_BASE}:mealkit-2024-01-13 .

@yhtang Yea, exactly. The changes from the TE team may not be in the main branch so they may wish to validate with rosetta, but at a different TE git-ref

@yhtang Bumping your review in case there's something else you'd like to discuss

repo: NVIDIA/JAX-Toolbox


text:
Armenian

"Upgraded" config for the Armenian MCV dataset
Fix various bugs in the sdp/processors
Add Contibution docs

Thanks, Ara!
We can't merge it, because it has a mix of pipeline and core changes.
Let's keep it simple and consider this branch only for Armenian-related stuff.

this brunch is under development

authors: Ara-Yeroyan karpnv

repo: NVIDIA/NeMo-speech-data-processor


text:
text2text.py data pipeline inefficiency

There are some inefficiencies with applying a shard after cache in text2text.py build_graph method.  Rewriting part of the code such that the shard is called first before map, cache results in a pretty sizable decrease in time per step when training big transformer on wmt14 en-de.  Happy to provide a pull request.  On 4 nodes, 32 gpus, batch size 128, iter size 16, mixed precision training, 16gb voltas, I've noticed time per step drop from 13.9s without the fix below to 6.6s with the following code change in the dataset pipeline.
  def build_graph(self):
    _sources = tf.data.TextLineDataset(self.source_file)

    if self._num_workers > 1:
      _sources = _sources.shard(num_shards=self._num_workers, index=self._worker_id)

    _sources = _sources.map(lambda line: tf.py_func(func=self._src_token_to_id, inp=[line],
                                   Tout=[tf.int32], stateful=False),
           num_parallel_calls=self._map_parallel_calls) \
      .map(lambda tokens: (tokens, tf.size(tokens)),
           num_parallel_calls=self._map_parallel_calls)

    _targets = tf.data.TextLineDataset(self.target_file) \

    if self._num_workers > 1:
      _targets = _targets.shard(num_shards=self._num_workers, index=self._worker_id)

    _targets = _targets.map(lambda line: tf.py_func(func=self._tgt_token_to_id, inp=[line],
                                   Tout=[tf.int32], stateful=False),
           num_parallel_calls=self._map_parallel_calls) \
      .map(lambda tokens: (tokens, tf.size(tokens)),
           num_parallel_calls=self._map_parallel_calls)

    _src_tgt_dataset = tf.data.Dataset.zip((_sources, _targets)).filter(
      lambda t1, t2: tf.logical_and(tf.less_equal(t1[1], self.max_len),
                                    tf.less_equal(t2[1], self.max_len))
    ).cache()

#204

Thanks @vsuthichai ! It does seem to make a big difference if horovod isn't used.
Is this what you meant: https://github.com/NVIDIA/OpenSeq2Seq/issues/243 ?
My only concern is whether shard is deterministic or not (it seems such) so that mapping between src and tgt is preserved

@okuchaiev Sorry, I didn't specify this earlier.  All experiments I've been doing are using Horovod, but yes, that appears to be the correct fix 👍

thx!
btw we changed license to Apache 2.0 so that it is easier for external people to send PRs https://github.com/NVIDIA/OpenSeq2Seq/blob/18.11-dev/LICENSE
(it currently shows MIT but this is because we haven't "released" 18.11 yet)

authors: okuchaiev vsuthichai


text:
Update TensorRT-LLM

Model Support

RoBERTa support (thanks to the contribution from @erenup)

Features

Chunked context support (see docs/source/gpt_attention.md#chunked-context)

API

Migrate models to the new build workflow, including LLaMA, Mistral, Mixtral, InternLM, ChatGLM, Falcon, GPT-J, GPT-NeoX, Medusa, MPT, Baichuan and Phi.
Deprecate LayerNorm and RMSNorm plugins
High-level-API: Support multi-GPU Tensor Parallelism on the LLM.generate_async API

Bug fixes

Fix INT8 GEMM shape #935

Thank you for the update!!!
We cam across a few questions
Are these argument now deprecated?
--remove_input_padding
--use_inflight_batching
--paged_kv_cache
More specifically, how much is changed from this tutorial
https://developer.nvidia.com/blog/optimizing-inference-on-llms-with-tensorrt-llm-now-publicly-available/
Thank You

i Run example/blip2 .Is the project missing convert_checkpoint.py files

Are these argument now deprecated? --use_inflight_batching

@enochlev Yes, the argument --use_inflight_batching was identical to --use_gpt_attention_plugin --use_paged_kv_cache --remove_input_padding in the previous version, for now, you can still enable inflight batching by enabling those three features, and note that they are enabled by default.
Please see the updated document, and use trtllm-build --help to get full help information. Thanks.

i Run example/blip2 .Is the project missing convert_checkpoint.py files

Hi @827346462 Please take a look and follow the document of blip2, we do not need convert_checkpoint.py for that model, thanks.

i Run example/blip2 .Is the project missing convert_checkpoint.py files

Hi @827346462 Please take a look and follow the document of , we do not need for that model, thanks.blip2``convert_checkpoint.py

follow the https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/blip2#guide-to-blip-2-pipeline  "python3 convert_checkpoint.py --model_dir ./opt-2.7b
--dtype float16
--output_dir ./opt/2.7B/trt_ckpt/fp16/1-gpu/"， in project no convert_checkpoint.py

authors: 827346462 enochlev kaiyux

repo: NVIDIA/TensorRT-LLM


text:
🐛[BUG]: IndexError: list index out of range when training BiStride MeshGraphNet

Version
0.8.0
On which installation method(s) does this occur?
Docker
Describe the issue
I'm trying to train a BiStride MeshGraphNet on my dataset (very similar to DrivAerNet), but I keep getting errors. It looks like it's expecting the data in the graph to have a very specific structure, unlike MeshGraphNet which is better written (and it trains on my data). The error I'm getting is
Traceback (most recent call last):
  File "/workspace/.../test_bsms_mgn.py", line 292, in <module>
    batch_loss = trainer.train(graph['graph'])
  File "/workspace/..../test_bsms_mgn.py", line 245, in train
    loss = self.forward(graph)
  File "/workspace/.../test_bsms_mgn.py", line 251, in forward
    pred = self.model(graph.ndata["x"], graph.edata["x"], graph)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1714, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1725, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/modulus/models/meshgraphnet/bsms_mgn.py", line 165, in forward
    x = self.bistride_processor(x, ms_ids, ms_edges, node_pos)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1714, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1725, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/modulus/models/gnn_layers/bsms.py", line 291, in forward
    h = self.down_gmps[i](h, m_gs[i], pos)
IndexError: list index out of range

Can you help? It would be useful if you provided an example to test BiStride MeshGraphNet out, but the only example mentioned in the documentation regards the Ahmed body dataset which is not included in the examples folder.
https://docs.nvidia.com/deeplearning/modulus/modulus-core/examples/cfd/aero_graph_net/readme.html#bsms-mgn-training
Minimum reproducible example
This is the dataset class:
class MyDataset(DGLDataset, Datapipe):
    def __init__(
        self,
        dir_list_file: str | Path,
        num_samples: int = None,
        invar_keys: Iterable[str] = ("pos", "X1", "X2"),
        outvar_keys: Iterable[str] = ("Y",),
        normalize_keys: Iterable[str] = None,
        cache_dir: str | Path = None, # "./cache/",
        force_reload: bool = False,
        name: str = "dataset",
        verbose: bool = False,
        triangulate: bool = True,
        downsampling_rate: int = 1,
        **kwargs,
    ) -> None:
        DGLDataset.__init__(self, name=name, force_reload=force_reload, verbose=verbose)
        Datapipe.__init__(self, meta=MetaData())

        with open(dir_list_file, 'r') as file:
            lines = [line.rstrip() for line in file]
        self.dir_list = [Path(f) for f in lines]
        for folder in self.dir_list:
            if not folder.is_dir():
                raise ValueError(
                    f"Path {folder} does not exist or is not a folder."
                )
        self.surface_filename = "surface.vtp"
        self.op_cond_json = "opcond.json"

        self.downsampling_rate = downsampling_rate
        self.triangulate = triangulate
        self.num_samples = num_samples
        self.input_keys = list(invar_keys)
        self.output_keys = list(outvar_keys)
        print(f"Input keys: {self.input_keys}")
        print(f"Output keys: {self.output_keys}")

        if normalize_keys:
            self.normalize_keys = list(normalize_keys)

        cache_dir_parent = self.dir_list[0].parent
        self.cache_dir = (
            self._get_cache_dir(cache_dir_parent, Path(cache_dir))
            if cache_dir is not None
            else None
        )

        list_op_cond = []
        for folder in self.dir_list:
            with open(folder / self.op_cond_json, "r") as fin:
                opc = json.load(fin)
            opc["folder"] = folder
            list_op_cond.append(opc)
        self.op_cond = pd.DataFrame(list_op_cond)
        self.op_cond.sort_values(by="folder", inplace=True)

        if self.num_samples:
            if self.num_samples > len(self.op_cond):
                raise ValueError(
                    f"Number of available {self.split} dataset entries "
                    f"({len(self.op_cond)}) is less than the number of samples "
                    f"({self.num_samples})"
                )
            self.op_cond = self.op_cond.iloc[:self.num_samples, ]

        numerical_df = self.op_cond.select_dtypes(include='number')
        normalized_df = (numerical_df - numerical_df.min()) / (numerical_df.max() - numerical_df.min())
        self.op_cond[numerical_df.columns] = normalized_df

    def __len__(self) -> int:
        return len(self.op_cond)

    def __getitem__(self, idx: int) -> dgl.DGLGraph:
        if not 0 <= idx < len(self):
            raise IndexError(f"Invalid {idx = }, must be in [0, {len(self)})")

        folder_path = self.op_cond.at[idx, "folder"]

        if self.cache_dir is None:
            graph = self._create_dgl_graph(folder_path, idx)
        else:
            cached_graph_filename = self.cache_dir / (folder_path.name + ".bin")
            if not self._force_reload and cached_graph_filename.is_file():
                gs, _ = dgl.load_graphs(str(cached_graph_filename))
                if len(gs) != 1:
                    raise ValueError(f"Expected to load 1 graph but got {len(gs)}.")
                graph = gs[0]
            else:
                graph = self._create_dgl_graph(folder_path)
                dgl.save_graphs(str(cached_graph_filename), [graph])

        graph.ndata["x"] = torch.cat([graph.ndata[k] for k in self.input_keys], dim=-1)
        graph.ndata["y"] = torch.cat([graph.ndata[k] for k in self.output_keys], dim=-1)

        return {
            "name": folder_path.name,
            "graph": graph,
            "X1": torch.tensor(self.op_cond.at[idx, "X1"], dtype=torch.float32),
            "X2": torch.tensor(self.op_cond.at[idx, "X2"], dtype=torch.float32),}

    @staticmethod
    def _get_cache_dir(data_dir, cache_dir):
        if not cache_dir.is_absolute():
            cache_dir = data_dir / cache_dir
        return cache_dir.resolve()

    def _create_dgl_graph(
        self,
        name: str,
        idx: int,
        to_bidirected: bool = True,
        dtype: torch.dtype | str = torch.int32,
    ) -> dgl.DGLGraph:

        def extract_edges(mesh: pv.PolyData) -> list[tuple[int, int]]:
            polys = mesh.GetPolys()
            if polys is None:
                raise ValueError("Failed to get polygons from the mesh.")

            polys.InitTraversal()
            edge_list = []
            for _ in range(polys.GetNumberOfCells()):
                id_list = vtk.vtkIdList()
                polys.GetNextCell(id_list)
                num_ids = id_list.GetNumberOfIds()
                for j in range(num_ids - 1):
                    edge_list.append(  # noqa: PERF401
                        (id_list.GetId(j), id_list.GetId(j + 1))
                    )
                # Add the final edge between the last and the first vertices.
                edge_list.append((id_list.GetId(num_ids - 1), id_list.GetId(0)))

            return edge_list

        surface_vtp_path = Path(name) / self.surface_filename

        surface_mesh = pv.read(surface_vtp_path)
        if self.triangulate:
            tmp_decimated_points = surface_mesh.points[::self.downsampling_rate,:]
            tmp_decimated_field = {}
            for target in self.output_keys:
                tmp_decimated_field[target] = surface_mesh[target][::self.downsampling_rate].reshape(-1,1)
            cloud = pv.PolyData(tmp_decimated_points)

            surface_mesh = cloud.delaunay_2d()
            for target in self.output_keys:
                surface_mesh[target] = tmp_decimated_field[target]

        edge_list = extract_edges(surface_mesh)

        graph = dgl.graph(edge_list, idtype=dtype)
        graph.ndata["pos"] = torch.tensor(surface_mesh.points, dtype=torch.float32)
        scalar_inputs = [k for k in self.input_keys if k != "pos" ]
        for k in scalar_inputs:
            graph.ndata[k] = torch.ones(surface_mesh.n_points, 1, dtype=torch.float32) * self.op_cond.loc[idx, k]

        for k in self.output_keys:
            graph.ndata[k] = torch.tensor(surface_mesh.point_data[k].reshape(-1, 1), dtype=torch.float32)

        u, v = graph.edges()
        pos = graph.ndata["pos"]
        disp = pos[u] - pos[v]
        disp_norm = torch.linalg.norm(disp, dim=-1, keepdim=True)
        graph.edata["x"] = torch.cat((disp, disp_norm), dim=-1)
        return graph

And this is the __init__ method of my trainer class:
class BSMGNTrainer:
    def __init__(self, cfg: DictConfig):
        self.dataset = MyDataset('/.../training_folders.txt',
                                          num_samples=cfg.num_samples, triangulate=cfg.triangulate, downsampling_rate=cfg.downsampling_rate,
                                          outvar_keys=cfg.target)
        self.dataloader = GraphDataLoader(self.dataset,
                                          shuffle=cfg.shuffle,
                                          batch_size=1,
                                          num_workers=cfg.num_workers,
                                          pin_memory=True,
                                          drop_last=True,)
        self.model = BiStrideMeshGraphNet(
            input_dim_nodes=len(self.dataset.input_keys) + 2,
            output_dim=len(self.dataset.output_keys),
            input_dim_edges= 4,
            mlp_activation_fn= 'relu',
            aggregation= 'sum',
            hidden_dim_processor=cfg.neurons,
            hidden_dim_node_encoder=cfg.neurons,
            hidden_dim_edge_encoder=cfg.neurons,
            hidden_dim_node_decoder=cfg.neurons)
        self.model = self.model.to(device)
        self.model.train()
        self.loss = torch.nn.L1Loss()
        self.optimizer = torch.optim.Adam(self.model.parameters(), lr=cfg.lr)
        self.scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer=self.optimizer, gamma=0.99985)

Relevant log output
No response
Environment details
No response

@Alexey-Kamenev could you please take a look?

You are correct, BSMS MGN expects the data in a certain format. To enable this format, you need to wrap your dataset class in BistrideMultiLayerGraphDataset like it's done in the Ahmed body example. You can do this either in the code or by using Hydra config - check out the BSMS Ahmed body experiment and corresponding dataset config.

I'm not sure I understand. Do you mean that, if I want to test both MeshGraphNet and BSMS MGN on the same data, I need to write two different dataset classes?

You don't need to write a new dataset class, all you have to do is to wrap your existing dataset class with BistrideMultiLayerGraphDataset class, like it's demonstrated in the config I mentioned in my response.
Specifically, in that config example, the already existing Ahmed Body dataset class, AhmedBodyDataset, is wrapped by BistrideMultiLayerGraphDataset. So in your case, all you have to do is provide your own, already existing, class instead of AhmedBodyDataset.
If you prefer doing it from the code rather than Hydra config, then the code will roughly look something like:
dataset = MyDataset(...)
if use_bsms:
    dataset = BistrideMultiLayerGraphDataset(dataset, num_layers=2, cache_dir="/data/bsms_l2_cache")

Closing the issue. Feel free to re-open or create a new one, if needed.

authors: Alexey-Kamenev AndreaPi mnabian

repo: NVIDIA/modulus


text:
[submodule-sync] bot-submodule-sync-branch-24.08 to branch-24.08 [skip ci] [bot]

submodule-sync to create a PR keeping thirdparty/cudf up-to-date.
HEAD commit SHA: 195a4e1cb0e5d8b0359aca403a8c607f23cbdc53, cudf commit SHA: https://github.com/rapidsai/cudf/commit/e6537de7474c91b4153542e6611c8a4e33a58caa
This PR will be auto-merged if test passed. If failed, it will remain open until test pass or manually fix.

HEAD commit SHA: 195a4e1cb0e5d8b0359aca403a8c607f23cbdc53, CUDF commit SHA: https://github.com/rapidsai/cudf/commit/e6537de7474c91b4153542e6611c8a4e33a58caa
Test passed: True

SUCCESS - auto-merge

repo: NVIDIA/spark-rapids-jni


text:
[auto-merge] branch-24.06 to branch-24.08 [skip ci] [bot]

auto-merge triggered by github actions on branch-24.06 to create a PR keeping branch-24.08 up-to-date. If this PR is unable to be merged due to conflicts, it will remain open until manually fix.

SUCCESS - auto-merge

repo: NVIDIA/spark-rapids-ml


text:
[FEA] Support Negative upper bound on current row in windowing

Is your feature request related to a problem? Please describe.
Exec <WindowExec> cannot run on GPU because not all expressions can be replaced
    !Expression <SpecifiedWindowFrame> specifiedwindowframe(my_range_frame, -12345, -1) cannot run on GPU because Upper-bounds behind current row is not supported. Found: -1

Describe the solution you'd like
Support upper bounds that can go behind the current row.
Describe alternatives you've considered
None
Additional context
Customer query

@kuhushukla which window operations is this for? I don't think it makes a lot of difference, but I want to make sure that we are testing the correct things, and it might reduce the total impact to the processing being done.
@mythrocks do you remember why we could not support this out of the box?  In practice I think we could replicate this is essentially doing a lead/lag like operation before the window operation. But I am not really 100% sure on that.

Sorry for the delay.  This took some recollecting.
TL;DR:  The CUDF primitives I've written do not support negative bounds.  It is doable with some non-trivial effort.
The primitives convert the range values into real row-offset values in the sort-by columns.  This was easy to do in the ROWS case.  In the RANGE case, the maths will be a little confusing.  I'd be delighted to work on it, depending on priority.

authors: kuhushukla mythrocks revans2

repo: NVIDIA/spark-rapids


text:
[BUG] Approx_percentile deserialize error when calling "show" rather than "collect"

Describe the bug
When running a show against an approx_percentile aggregate query, the query fails with a serialization error due to the partial aggregate running on GPU and the final aggregate running on CPU. The intermediate aggregate buffer is not in the expected format for CPU so deserialization fails.
The final stage falls back to the CPU due to a cast to string that is trigger by show and is not supported on GPU.
!Expression <Cast> cast(approx_percentile(v#476, [0.1,0.2], 10000, 0, 0)#479 as string) cannot run on GPU because Cast from ArrayType(DoubleType,false) to StringType is not supported

Example of the error stack trace:
Caused by: java.lang.NegativeArraySizeException
at org.apache.spark.sql.catalyst.expressions.aggregate.ApproximatePercentile$PercentileDigestSerializer.deserialize(ApproximatePercentile.scala:319)

Steps/Code to reproduce bug
I can reproduce with this integration test (added to hash_aggregate_test.py).
@pytest.mark.parametrize('aqe_enabled', ['false', 'true'], ids=idfn)
@ignore_order(local=True)
@allow_non_gpu('TakeOrderedAndProjectExec', 'Alias', 'Cast', 'ObjectHashAggregateExec', 'AggregateExpression',
    'ApproximatePercentile', 'Literal', 'ShuffleExchangeExec', 'HashPartitioning', 'CollectLimitExec')
def test_hash_groupby_approx_percentile_double_show(aqe_enabled):
    conf = copy_and_update(_approx_percentile_conf, {
        'spark.sql.adaptive.enabled': aqe_enabled,
        'spark.rapids.sql.explain': 'ALL'
    })

    def foo(spark):
        df = gen_df(spark, [('k', StringGen(nullable=False)),
                                         ('v', DoubleGen())], length=100)
        df.createOrReplaceTempView("t")
        df2 = spark.sql("SELECT k, approx_percentile(v, array(0.1, 0.2)) from t group by k")

        # this "show" triggers the exception
        df2.show()

        return df2

    run_with_cpu_and_gpu(foo, 'COLLECT', conf)

Expected behavior
We should recognize that the final aggregate cannot run on GPU and force the partial aggregate to fall back to CPU as well.
Environment details (please complete the following information)
N/A
Additional context
N/A

Please make sure that for any test in the future that we have some way to permanently force the second half onto the CPU.  It is very likely that we will support casting arrays to strings in the future.

There are checks already that tag the partial aggregate to indicate that it should fall back to the CPU but then this tag was not being checked when planning the query stage containing the partial aggregate. I was able to confirm this by adding the following code to GpuTypedImperativeSupportedAggregateExecMeta.tagPlanForGpu.
override def tagPlanForGpu(): Unit = {
  super.tagPlanForGpu()

  // when AQE is enabled and we are planning a new query stage, we need to look at meta-data
  // previously stored on the spark plan to determine whether this aggregate can run on GPU
  wrapped.getTagValue(gpuSupportedTag).foreach(_.foreach(willNotWorkOnGpu))

However, it shouldn't be necessary to add this check on each meta class. We should be able to walk the plan during query stage planning to make sure this gets invoked on all nodes.


text:
[BUG] Markdown link check always fails during release process

The markdown link check on premerge keeps failing after we update the download links before we release.  It's a chicken-and-egg problem where we want to update the docs before tagging the final build candidate, but we don't want to release the final build candidate until all of the final QA testing has completed.  During that final testing of the release candidate, the markdown link keeps failing on that dev branch as well as the next release's dev branch (due to automerge of the download page update).
The problem with markdown link checks always failing is that developers stop looking into any failures because it routinely fails during the release process.  Ideally we should get to the point where these are not always failing and thus a failure would be rare and interesting to investigate.  One idea is to have the markdown link check be aware of release download links (i.e.: links to a RAPIDS Accelerator jar in Maven Central) and not complain about the link if it is for a build version that has not officially released yet.  Ideas for checking if a build version has been released include:

checking the list of tags on the repo to see if the version has a corresponding tag
checking the Github Pages version of the download doc and seeing if the same release download link appears there
If the build version does not appear to be released, the markdown link check would not complain about that link.

I checked the markdown link checker document and seems it does not support customized conditions. And if we just disable checks for dev branch and there is a real link issue, we cannot find it in time. How about this:
We just ignore these maven links(add two ignore-patterns: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/22.08.0/*   and https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/22.08.0/) before release, and after we final release, I just draft a pr to update the ignore patterns(update two ignore-patterns to: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/22.10.0/   and https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/22.10.0/*).

We just make download link updates of plugin directly in to gh-pages

I like where @pxLi is going with this, but I think there should be some refinements:

We should update the links as part of the merge to main PR.  IMO the gh-pages should always come from main, not some custom changes directly on gh-pages
We need to update the download links in the next dev branch after the previous dev branch has been released.  People can visit our github page, which defaults to the latest dev branch, and it would be nice if the download page docs they browse there have links to the released artifacts.

And if we just disable checks for dev branch and there is a real link issue, we cannot find it in time.

To clarify, I don't think anyone is proposing we disable link checks on a dev branch wholesale.
I think either idea works (only update main/gh-pages during release, or ignore download links during release).  I have a slight preference for the former, as I think it's less PRs overall to achieve the desired effect.

@jlowe I also prefer the former solution. It means each time, I did not put the download links in dev branch.
Instead, @pxLi 's team to update the main branch with the new download links during release time.
Once main is updated, @nvliyuan  can update the gh-pages based on the difference between main and gh-pages branches.
Sounds good to everybody?

Sounds good to everybody?

yes, already sync with @NvTimLiu

I added a step in the 22.10 release checklist https://github.com/NVIDIA/spark-rapids/issues/6255 for creating a PR to update the release links in the main branch.

I also added a step to the 22.10 release checklist to create a PR to update the next development branch with the links to the previous release.

Is this a duplicate of https://github.com/NVIDIA/spark-rapids/issues/10834 ?

Yes.

authors: jlowe nvliyuan sameerz viadea


text:
Add DateType support for AST expressions

Signed-off-by: remzi 13716567376yh@gmail.com
close #3752
This PR adds DateType to AST expressions

build


text:
Adds link to spark supporting shuffle classes and fix copyright

This PR does minor follow up as this comment states: https://github.com/NVIDIA/spark-rapids/pull/6511#discussion_r975458436.
It also fixes a missing copyright that was committed in the same PR.

build


text:
Fix leak in BatchWithPartitionData

We see a leak in BatchWithPartitionDataSuite but that led me to fix another issue that seemed worse in BatchWithPartitionData, where any time there is a split + retry we are leaking a batch.
Hope this makes sense @parthosa, I'd appreciate you taking a look since you are most familiar with the code. Thanks!

build


text:
What is the algorithm delay of WaveGlow

If I understand correctly, even though the WaveGlow has a parallel generative framework, it still need to take sufficient amount of speech samples before it can generate audio sample. Then what would be the algorithm delay of WaveGlow? If my input audio segment length is 16000, as the default value in config.json. Does it translate to the delay of WaveGlow is 16000 samples? @rafaelvalle

But as I understood, the inference starts with random sampling from z. Previous speech samples are not required.
Coming to the 16000 samples, as mentioned in the paper, the training is done using randomly chosen clips of 16000 samples. Probably the default value is for the wave chunks in the training phase.
The algorithm delay is also my question. I am also confused about the receptive field size. with 8 dilations (1, 2, ..., 128), filter_width=3 (taps), the receptive filed size is 511, according to rf=sum((fw-1)*dilation(i))+1.
Does groups of 8, in squeeze operation, multiplies the receptive field size by 8?

Closing due to inactivity.

I have done some investigations about the algorithm delay and the receptive fields.
The first thing is that the algorithm delay comes from the mel spectrogram, and when the inference begins, the mel is upsampled to the same size of audio and then split into 8 groups. So for 1 mel frame, it is upsampled to 256 timesteps, and becomes 32 (256/8) timesteps for each group.
So the 511 steps of delay observed by @AzamRabiee inside the wavenet structure would represent an 8(511/32)-frame delay.
However, I am somewhat confused by the receptive field of the entire waveglow. As I mentioned, every 1 flow would have 8 frames of delay, so that means, theoretically, the accumulated algorithm delay of waveglow should be 8 x 12flows = 96 frames. Also the receptive field should be 96*2+1 frames. But, the algorithm delay I observed in my experiment is only 16 frames, which is very weird.
Right now, I am not quite sure how waveglow reduces the accumulated algorithm delay.

authors: AzamRabiee cjerry1243 rafaelvalle triwoods

repo: NVIDIA/waveglow


text:
Render Different viewpoints at once

Thanks for creating this wonderful package!
Is there a way to render multiple viewpoints of an object  without a for loop to iterate through and
set different camera positions? The current examples in SoftRas indicate that it is possible. What about DIB-R and NMR?
Thanks

Currently, the DIB-R renderer does support rendering from multiple views (i.e. batched camera parameters). The base renderer simply wraps around a specific renderer (for a particular rendering mode) and sets a single camera parameter. For now, to pass batched camera parameters, use one of the specific renderer class (e.g. kaolin.graphics.dib_renderer.renderer.VCRenderer) instead of the base renderer (kaolin.graphics.DIBRenderer).
Note that currently only camera position and rotation parameters are batched; the camera projection matrix is still shared. However, it's not hard to refactor the code to support the batched projection matrix as well if the need arises.

DIB-R does support some form of camera parameter batching. Each camera parameter (e.g. azimuth, elevation) is a list, and the number of elements you pass in must equal the number of meshes you give (i.e. the shape of the first dimension for vertices). To render the same mesh from different viewpoints, you need to use expand on the vertices to match the number of camera parameters you pass in.
Note that only camera position and rotation parameters are batched; the camera projection matrix is still shared. From the looks of it, it's not hard to refactor the code to support the batched projection matrix as well if the need arises.

Closing for inactivity, it's now completely possible to transform a mesh through multiple cameras and render the viewpoints as a batch

authors: Caenorst TommyX12 vaibhavnayel

repo: NVIDIAGameWorks/kaolin


text:
Assassins creed 2

Describe the bug
game crashes on boot up
GPU:3070TI
Game: Assassins Creed 2 (Ubisoft Connect)
latest version of remix
Attach logs!
AssassinsCreedIIGame_d3d9.log
d3d9.log
NvRemixBridge.log
To Reproduce
boot game from exe
Expected behavior
boot to main menu

Describe the bug game crashes on boot up
GPU:3070TI Game: Assassins Creed 2 (Ubisoft Connect) latest version of remix Attach logs! AssassinsCreedIIGame_d3d9.log d3d9.log NvRemixBridge.log
To Reproduce boot game from exe
Expected behavior boot to main menu

you need to bypass the video cutscenes with eizo talking with the text in the room wall and the artimis. just make blank bik files

Hello! Will you please test this on the latest release and let us know if the issue persists? Thank you!

authors: NV-LL Rmurray2006 howdoicodeplease

repo: NVIDIAGameWorks/rtx-remix


text:
The Stanley Parable 2013 - Remix doesn't start

I was trying to use Remix with the original "The Stanley Parable" game from Steam.
No splash screen with "Welcome to NVIDIA Remix…" appeared and alt+x doesn't do anything.
Looks like Remix doesn't launch at all.
Info

Remix version: remix-0.2.0
GPU: GeForce RTX 4090
GPU driver version: 537.42
Game: The Stanley Parable - Steam - original version from 2013

Logs
I didn't find any useful logs unfortunately
To Reproduce

Download The Stanley Parable game from Steam
Press "Manage" -> "Browse Local Files"
Add all the files from remix-0.2.0
Launch the game
Observe no splash screen with welcome message "Welcome to NVIDIA Remix…"
Try alt+x hotkey, observe it doesn't work

There are a few ways to launch a game:

From the Steam menu
From the game folder using stanley.exe, for this you need to create a shortcut with flags:

-game thestanleyparable - to specify which game folder should be used
-dxlevel 90 (optional) - to specify DirectX version, expected supported values are: 60, 70, 80, 81, 90, 91, 95, 98 and 100

Expected behavior

After game is launched splash screen with welcome message "Welcome to NVIDIA Remix…" should appear.
Hotkey alt+x should open Remix control panel

Actually, it doesn't work for any games I tried, including:

Portal
Half-Life 2

I am following the runtime-user-guide by doing this step:

To prep the game, simply copy the contents of the remix-0.1.0 directory into the game’s directory, right next to the main executable of the game. You should end up with the d3d9.dll and .trex/ folder sitting right next to the main game executable.
Once RTX Remix files are in place, you can start the game normally. You can verify RTX Remix is working by checking for the splash message at the top of the screen when the game starts. It should say: “Welcome to NVIDIA Remix… ” and provide hotkey information to access the Remix menus.

The only overlay welcome message I can see is "GeForce Experience" with alt+z hotkey, which is working.
This is how "The Stanley Parable" game folder file tree looks like with added "remix-0.2.0" content:

Okay... looks like some files should actually be placed inside bin folder for some reason. Could runtime-user-guide be updated to reflect where in which cases files should be places, to avoid confusion?

@nv-ajaus Could you take a look please?

Hi @Friend-LGA , thanks for bringing this up! Just to clarify, when you installed rtx-remix to the bin folder, did The Stanley Parable + Remix work for you? I'll bring this up with the team to see how we could improve the documentation.

I've added a note to the Runtime User Guide for this installation case.

authors: Friend-LGA NV-DM nv-ajaus


text:
How to introduce RGBD image to the policy network for the handover_sim environment

I want to obtain the RGBD as state and input to the policy network, but I find the simulator rendering became very slow. I mainly modify three prats of the code.
Specifically, I modify part of the code in the project, including handover_env.py (class HandoverHandCameraPointStateEnv(HandoverEnv) ) and run_benchmark_gaddpg_hold.py (class GADDPGPolicy(SimplePolicy) and class PointListener).

I want to introduce the rgb and depth image to the network, so add two more items to the observation, i.e., observation["rgb"] and observation["depth"], as following:

class HandoverHandCameraPointStateEnv(HandoverEnv) in handover_env.py with modification details:
def _get_observation(self):
    observation = {}
    observation["frame"] = self.frame
    observation["panda_link_ind_hand"] = self.panda.LINK_IND_HAND
    observation["panda_body"] = self.panda.body
    observation["callback_get_point_state"] = self._get_point_state
    **observation["rgb"]=self.panda._camera.color[0]
    observation["depth"]=self.panda._camera.depth[0]**
    return observation

Use obs["rgb"] and obs["depth"] to the policy network, with corresponding modification details:

class GADDPGPolicy(SimplePolicy) in run_benchmark_gaddpg_hold.py
def plan(self, obs):
    info = {}

    if (obs["frame"] - self._steps_wait) % self._steps_action_repeat == 0:
        point_state, obs_time = self._get_point_state_from_callback(obs)
        info["obs_time"] = obs_time

        if point_state.shape[1] == 0 and self._point_listener.acc_points.shape[1] == 0:
            action = np.array(self._cfg.ENV.PANDA_INITIAL_POSITION)
        else:
            ef_pose = self._get_ef_pose(obs)
            ef_pose_new = self._point_listener.run_network(**(point_state, obs["rgb"], obs["depth"])**, ef_pose)
       ...
       ...

rgb and depth concat together and resize to (112, 112), with corresponding modification details:

class PointListener in run_benchmark_gaddpg_hold.py with modification details::
def _point_state_to_state(self, point_state, ef_pose):
point_state, rgb, depth = point_state[0],point_state[1],point_state[2]
rgb=rgb.numpy()[:,:,:3]
depth = depth.numpy()
depth=depth[:,:,np.newaxis]
rgbd=np.concatenate([rgb,depth],axis=2)
rgbd=np.transpose(rgbd, (2,0,1))
rgbd_resized=np.zeros((4,112,112))
for i in range(rgbd.shape[0]):
rgbd_resized[i]=cv2.resize(rgbd[i],(112,112))
    point_state = self._process_pointcloud(point_state, ef_pose)
    # image_state = np.array([])
    image_state=rgbd_resized
    obs = (point_state, image_state)
    return obs

Is there something wrong with the modification codes?
I would like to know how to introduce RGBD image to the policy network for the handover_sim environment.

Thank you for your detailed explanation. I learnt a lot and appreciated the thoughtful design in the code.
I followed your suggestion and the code now works fine.

repo: NVlabs/handover-sim


text:
Create 2010-2020 and 2020 geo_lookups

2020
Using DRAFT BYTES shapefiles, will update with published data eventually.
WITH
blocks AS (
	SELECT
		geoid,
		bctcb2020,
		RIGHT(bctcb2020, 10) as ctcb2020,
		SUBSTRING(bctcb2020, 2, 7) as ctcbg2020,
		ct2020,
        borocode,
        boroname,
        (CASE
        	WHEN LEFT(geoid, 5) = '36061' THEN 'New York'
        	WHEN LEFT(geoid, 5) = '36005' THEN 'Bronx'
        	WHEN LEFT(geoid, 5) = '36047' THEN 'Kings'
        	WHEN LEFT(geoid, 5) = '36081' THEN 'Queens'
        	WHEN LEFT(geoid, 5) = '36085' THEN 'Richmond'
        END) as county,
        LEFT(geoid, 5) as county_fips,
        ST_Centroid(wkb_geometry) as cb2020_centroid_geom
	FROM census_2020_draft.cb2020
),
tracts AS (
	SELECT
		a.geoid,
		a.bctcb2020,
		b.boroct2020,
		a.ctcb2020,
		a.ctcbg2020,
		a.ct2020,
		b.nta2020,
		b.ntaname,
		b.cdta2020,
		b.cdtaname,
		a.borocode,
		a.boroname,
		a.county_fips,
		a.county,
		a.cb2020_centroid_geom
	FROM blocks a
	JOIN census_2020_draft.ct2020 b
	ON a.borocode||a.ct2020 = b.boroct2020
),
fp_500 AS (
	SELECT
		a.bctcb2020,
		ST_Intersects(a.cb2020_centroid_geom, b.wkb_geometry)::int as fp_500
	FROM tracts a, geo_lookups.flood_500 b
),
fp_100 AS (
	SELECT
		a.bctcb2020,
		ST_Intersects(a.cb2020_centroid_geom, b.wkb_geometry)::int as fp_100
	FROM tracts a, geo_lookups.flood_100 b
),
park_union AS (
	SELECT ST_Union(wkb_geometry) as geom
	FROM  dpr_access_zone.latest
),
park_access AS (
	SELECT
		a.bctcb2020,
		ST_Intersects(a.cb2020_centroid_geom, b.geom)::int as park_access
	FROM tracts a, park_union b
)
SELECT
	a.geoid2020,
	a.bctcb2020,
	a.boroct2020,
	a.ctcb2020,
	a.ctcbg2020,
	a.ct2020,
	a.nta2020,
	a.ntaname,
	a.cdta2020,
	a.cdtaname,
	a.borocode,
	a.boroname,
	a.county_fips,
	a.county,
	fp_500.fp_500,
	fp_100.fp_100,
	park_access.park_access
INTO census_2020_draft.geo_lookup
FROM
	tracts a
LEFT JOIN fp_500 ON a.bctcb2020 = fp_500.bctcb2020
LEFT JOIN fp_100 ON a.bctcb2020 = fp_100.bctcb2020
LEFT JOIN park_access ON a.bctcb2020 = park_access.bctcb2020
;

2010 to 2020
SELECT
	b.ctcb2010 as geoid2010,
	a.geoid as geoid2020,
	a.bctcb2020,
	a.boroct2020,
	a.ctcb2020,
	a.ctcbg2020,
	a.ct2020,
	a.nta2020,
	a.ntaname,
	a.cdta2020,
	a.cdtaname,
	a.borocode,
	a.boroname,
	a.county_fips,
	a.county
FROM census_2020_draft.geo_lookup a
JOIN census_cb10_cb20 b
ON a.geoid = b.ctcb2020;

#108

#214

authors: SPTKL mgraber

repo: NYCPlanning/db-factfinder


text:
Store caches errors

I'm not sure if this is intended behavior. If an Observable errors, subsequent calls to get just repeat that error.
I would expect the store to retry after an error.
public class DontCacheErrorsTest {

    private boolean shouldThrow;
    private Store<Integer> store;

    @Before
    public void setUp() {
        store = StoreBuilder.<Integer>builder()
                .fetcher(barCode -> Observable.fromCallable(() -> {
                    if (shouldThrow) throw new RuntimeException();
                    else return 0;
                }))
                .open();
    }

    @Test
    public void testStoreDoesntCacheErrors() throws InterruptedException {
        BarCode barcode = new BarCode("bar", "code");

        shouldThrow = true;
        store.get(barcode).test()
                .awaitTerminalEvent()
                .assertError(Exception.class);

        shouldThrow = false;
        store.get(barcode).test()
                .awaitTerminalEvent()
                .assertNoErrors();
    }
}

Agree on that issue, had stumble upon it before but did not have the time to open it. Retrying though is kind of obtrusive. I would expect that the Store would not write a resource and just hit the fetcher again.

Issue is due to inflight being unaware if it is a good or error response and caching it for 1 min.  I'll make a change that will clear inflight if an error is read.  I don't believe any additional retry logic is needed.  Thanks for reporting PR coming....

I was a little unclear. With retry I mean that in like in my test the second get should call the fetcher again.

authors: PaulWoitaschek digitalbuddha pavlospt

repo: NYTimes/Store


text:
Bump gcloud SDK / kubectl to latest

This is to facilitate apply actions against PodSpecs that have HostAlias fields (new in 1.7).

@tonglil should I close this in light of PR72?

repo: NYTimes/drone-gke


text:
Need the ability to update a wishlist

As a customer
I need to update a wishlist
So that I can add a new product to it.
Assumptions

customer_id is given, this refers to a customer on the customers microservice
wishlist_id which already exists is given
product_id is given, this refers to a product on the products microservice

Acceptance Criteria
Given the customer_id , wishlist_id and product_id
When the request for update is received
Then the product_id is added to the wishlist

Should also be able to update the name of the wishlist

Hey team! Please add your planning poker estimate with ZenHub @AashiqMd @MDShahedurRahman

Please add your planning poker estimate with ZenHub @neerajkanuri

Please add your planning poker estimate with ZenHub @anugyashah01

authors: AashiqMd neerajkanuri


text:
.xml 谱面的导出精度限制为 1e-6 以适配 Dym

观察从 Dym(Jmak) 中导出的谱面精度并非限制为 1e-6
需要更多信息

从目前看来没有这么做的必要。
包括 DyM-modified 自己导出的xml谱面都没有限制为1e-6。

repo: NagaseIori/DyNode


text:
Please add an analogue of emplace

@Naios

Being able to construct an object rather than having to copy/move one in would be very useful, particularly for objects that may be expensive to construct, or (for unique_function) immovable objects. (After all, if they're on the heap we can still move the pointer.)

Commit Hash
7cd95374b0f1c941892bfc40f0ebb6564d33fdb9
Expected Behavior
Being able to type something like
fun.emplace<Foo>(something, something)
Actual Behavior
Needing to type fun = Foo(something, something)
Steps to Reproduce
N/A
Your Environment

OS: Fedora 30
Compiler and version: GCC 9.2.1
Standard library (if non default): libstdc++-9.2.1

This sounds like a promising feature.
Probably I will implement it in the future.

authors: Naios adamemerson


text:
OnError pipeline exception is innermost exception

For my error handling, I have defined a global exception handler that catches exceptions on the on error handling. As of version 1.3 - any exceptions with an inner exception (no matter how many inner exceptions) are only sent through the pipeline as the innermost exception.
Is this expected behavior?

This sounds like a dupe of #1638

authors: ekelundh khellang

repo: NancyFx/Nancy


text:
add support to class cards

With the new expansion "forgotten realms", they added a new type of cards: class cards. This is one example
cleric class json response and cleric class scryfall view.
So basically I added a one-liner to support them. Can you please merge the PR?
Thanks! 😄

Even tough I'm on 1.9.0, I still can't get the image_uris() for class cards.
I'm on python 3.9.1, even tried uninstalling the one I got from pip and reinstalling the git repo version

repo: NandaScott/Scrython