text:
Timer stats

Tickets addressed: N/A
Review: By commit
Merge strategy: Merge (no squash)

Description
Logs wall clock time and CPU time performance of simulation, scheduling, separating out database writing and, optionally via Java property, calls to the adaptation.   Here is an example of what appears in the combined logs for simulation and scheduling when grepping with egrep '([|] Request: POST)|Timer':
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer - property gov.nasa.jpl.aerie.timeTasks = null
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer - Timer.timeTasks = false
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer - average time of system call = 674 nanoseconds
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:14.014562 -- server getSimulationResults() -- start
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:14.082387 -- server getSimulationResults() -- end
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:14.084937 %% REPORTING TIMER STATS %%
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer -  %% server getSimulationResults(): start = 2023-030T20:12:14.014562
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer -  %% server getSimulationResults(): duration = 0.067824208 seconds
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer -  %% server getSimulationResults(): cpu time = 0.021897811 seconds
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer -  %% server getSimulationResults(): end = 2023-030T20:12:14.082387
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer -  %% start server getSimulationResults(),end server getSimulationResults(),cpu time server getSimulationResults(),wall clock time server getSimulationResults(),count server getSimulationResults()
aerie_merlin              | [qtp1764996806-48] INFO gov.nasa.jpl.aerie.stats.Timer -  %% 1675109534014562996,1675109534082387204,21897811,64218208,1
aerie_merlin              | Request: POST [/getSimulationResults]
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer - property gov.nasa.jpl.aerie.timeTasks = null
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer - Timer.timeTasks = false
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer - average time of system call = 743 nanoseconds
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:14.262433 -- worker simulation -- start
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:14.686763 -- worker simulation -- end
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:14.68708 %% REPORTING TIMER STATS %%
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% worker simulation: start = 2023-030T20:12:14.262433
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% worker simulation: duration = 0.424330046 seconds
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% worker simulation: cpu time = 0.342728728 seconds
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% sim without writing: start = 2023-030T20:12:14.327363
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% sim without writing: duration = 0.149750657 seconds
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% sim without writing: cpu time = 0.138953591 seconds
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% sim without writing: end = 2023-030T20:12:14.477113
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write sim to DB: start = 2023-030T20:12:14.477189
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write sim to DB: duration = 0.209505256 seconds
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write sim to DB: cpu time = 0.152690132 seconds
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write sim to DB: end = 2023-030T20:12:14.686694
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% worker simulation: end = 2023-030T20:12:14.686763
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% start sim without writing,end sim without writing,cpu time sim without writing,wall clock time sim without writing,count sim without writing,start worker simulation,end worker simulation,cpu time worker simulation,wall clock time worker simulation,count worker simulation,start write sim to DB,end write sim to DB,cpu time write sim to DB,wall clock time write sim to DB,count write sim to DB
aerie_merlin_worker_2     | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% 1675109534327363245,1675109534477113902,138953591,149625681,1,1675109534262433159,1675109534686763205,342728728,420952393,1,1675109534477189559,1675109534686694815,152690132,209477743,1
aerie_scheduler           | [qtp231202600-39] INFO gov.nasa.jpl.aerie.stats.Timer - property gov.nasa.jpl.aerie.timeTasks = null
aerie_scheduler           | [qtp231202600-39] INFO gov.nasa.jpl.aerie.stats.Timer - Timer. timeTasks = false
aerie_scheduler           | [qtp231202600-39] INFO gov.nasa.jpl.aerie.stats.Timer - average time of system call = 735 nanoseconds
aerie_scheduler           | [qtp231202600-39] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:24.888831 -- server schedule() -- start
aerie_scheduler           | [qtp231202600-39] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:25.033334 -- server schedule() -- end
aerie_scheduler           | Request: POST [/schedule]
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer - property gov.nasa.jpl.aerie.timeTasks = null
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer - Timer.timeTasks = false
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer - average time of system call = 763 nanoseconds
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:25.121622 -- worker scheduling -- start
aerie_scheduler           | [qtp231202600-34] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:25.602738 -- server schedule() -- start
aerie_scheduler           | [qtp231202600-34] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:25.634673 -- server schedule() -- end
aerie_scheduler           | Request: POST [/schedule]
aerie_merlin              | Request: POST [/resourceTypes]
aerie_scheduler           | [qtp231202600-36] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:26.16047 -- server schedule() -- start
aerie_scheduler           | [qtp231202600-36] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:26.202649 -- server schedule() -- end
aerie_scheduler           | Request: POST [/schedule]
aerie_scheduler           | [qtp231202600-38] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:26.727154 -- server schedule() -- start
aerie_scheduler           | [qtp231202600-38] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:26.747124 -- server schedule() -- end
aerie_scheduler           | Request: POST [/schedule]
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:27.10201 -- worker scheduling -- end
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:27.102381 %% REPORTING TIMER STATS %%
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% worker scheduling: start = 2023-030T20:12:25.121622
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% worker scheduling: duration = 1.980388138 seconds
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% worker scheduling: cpu time = 0.553469331 seconds
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% collect scheduling results: start = 2023-030T20:12:27.074642
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% collect scheduling results: duration = 0.003475532 seconds
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% collect scheduling results: cpu time = 0.002367103 seconds
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% collect scheduling results: end = 2023-030T20:12:27.078118
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write scheduling results: start = 2023-030T20:12:27.078362
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write scheduling results: duration = 0.023618686 seconds
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write scheduling results: cpu time = 0.009277075 seconds
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write schedule to DB: start = 2023-030T20:12:27.078578
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write schedule to DB: duration = 0.0233293 seconds
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write schedule to DB: cpu time = 0.008996446 seconds
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write schedule to DB: end = 2023-030T20:12:27.101907
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% write scheduling results: end = 2023-030T20:12:27.10198
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% worker scheduling: end = 2023-030T20:12:27.10201
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% start collect scheduling results,end collect scheduling results,cpu time collect scheduling results,wall clock time collect scheduling results,count collect scheduling results,start worker scheduling,end worker scheduling,cpu time worker scheduling,wall clock time worker scheduling,count worker scheduling,start write schedule to DB,end write schedule to DB,cpu time write schedule to DB,wall clock time write schedule to DB,count write schedule to DB,start write scheduling results,end write scheduling results,cpu time write scheduling results,wall clock time write scheduling results,count write scheduling results
aerie_scheduler_worker_2  | [main] INFO gov.nasa.jpl.aerie.stats.Timer -  %% 1675109547074642504,1675109547078118036,2367103,3291900,1,1675109545121622682,1675109547102010820,553469331,1974841029,1,1675109547078578101,1675109547101907401,8996446,23107624,1,1675109547078362054,1675109547101980740,9277075,23456883,1
aerie_merlin              | Request: POST [/validateActivityArguments]
aerie_merlin              | Request: POST [/getActivityEffectiveArguments]
aerie_scheduler           | [qtp231202600-36] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:27.267125 -- server schedule() -- start
aerie_scheduler           | [qtp231202600-36] INFO gov.nasa.jpl.aerie.stats.Timer - 2023-030T20:12:27.320005 -- server schedule() -- end
aerie_scheduler           | Request: POST [/schedule]

The basic structure of logging stats for simulation and scheduling is roughly the same.  This is the structure for simulation:

timestamp when sim request on the server starts
timestamp when sim request on the server ends
list of stats for the server request
csv header for server stats
csv row for server stats   -- for  pasting in a spreadsheet, for example, to more easily visualize and compare the stats
timestamp when worker simulation starts
timestamp when worker simulation ends
list of stats for the worker simulation

ordered start and end times of different parts of the simulation (sim before DB, writing to DB, and adaptation)
end times of parts are followed by their list of stats

CSV header for worker sim stats
CSV row for worker sim stats

This list is logged each time simulation is invoked.
Logged once during static initialization of the TImer class are the java property value and the overhead of the system time call to get nanoseconds since 1/1/1970.  This overhead is used to correct the wall clock time.  In the case above, the Java property was left out.
Verification
Build tests passed.  I verified logging with the adaptation part turned on and off and left out.
The wall clock time for the adaptation was inconsistent.  The adaptation calls each take 1 ms on average, so there appears to be error at the level of a millisecond, possibly due to the timer code skewing results.  Since this level of timing appears to be too invasive, it is turned off by default.  More testing is planned to verify suspicions.
Documentation
I didn't update documentation, but if there is an appropriate place for it, please let me know, and I'll add it.  We probably don't document how to look at the log files, but the Java property to turn on the timing adaptation calls could be discussed.
Future work
None planned, but kind of wanted to write out html to visualize results; did the CSV instead based on user feedback.

Regarding the wall clock mismatch mystery, I did some follow-up testing by adding sleeps in the adaptation code. There is occasional interleaving/concurrency of adaptation calls that explains why the summed wall clock time of the adaptation calls is sometimes longer than the time period from the start of the first call to the end of the last.  It doesn't appear to be due to timing error.  Keeping the measurement of adaptation calls turned off by default still makes sense because there will still be overhead even if there is no error.

Instead of injecting test code into Aerie, could you instead create performance tests?

This is just a random drive-by thought, but could we use the Instrumentation API to Instrument the simulation system from the outside?

Instead of injecting test code into Aerie, could you instead create performance tests?

I expect to create performance tests, and I expect I'll want to use the logging to collect results.  I am under the impression that users like David Legg, Ansel, and Parker, will appreciate it, too.  With the default of not measuring the adaptation code, I don't know what the disadvantage would be.

What I'm requesting is rather than injecting this test code into Aerie, you create tests that call the Aerie methods and have this benchmarking code in the tests.

Yes, I think performance tests are valuable and we should have more (any) of them.
On the other hand, I think that the fine-grained timings Brad is hunting for are useful, and hard to get at from a top-level performance test. At the very least, they could be useful for identifying parts of the system where most time is being spent, so we can target optimizations to those areas.
On the gripping hand, I strongly agree with @Mythicaeda that we shouldn't inject test/performance code directly into our codebase if we can help it. It serves a purpose distinct from the code around it, and can both add noise to the logic it intrudes upon and fall out of date/usefulness as the code around it shifts.
That's why I suggested the Instrumentation API. We can, in principle, inject this kind of performance code into the system, at targeted points like Brad has done, but at classload time, so that we don't pollute the codebase itself with this concern. This is a limited form of aspect-oriented programming.

I guess I'm used to instrumenting my code directly with performance stat collectors that it never seemed intrusive or inappropriate to me.  You always want to log this information: in development, test, and production.  It's most useful for dev ops.
I like the example with the instrumentation API.  This is a good case for the aspect-oriented approach.  I think that will work, even around calls to the adaptation.  A disadvantage to using this is that it adds another layer of complication to the software, making it harder to understand and maintain (and me to develop).
But, if y'all prefer the instrumentation approach, I can do that.

I'm personally less concerned about intrusion of performance hooks at high-level control points, especially at service boundaries. I haven't taken a close look, but the timings you're tracking seem to be of that kind.
However, I think that the instrumentation API will be something we can leverage for far more things, including probing timings within the simulation engine itself. We absolutely don't want to pollute that with excess performance scaffolding, because the logic it would intrude on is already complex enough. I think the Instrumentation API would be ideal for making those kinds of measurements.
In sum: not too worries about the kinds of timings you're measuring, but using the Instrumentation  API now will give us a platform to build on for much more interesting and fine-grained timings as well. If it's something you are interested in doing, I think it would position us very well.

Good point--I knew there were other benefits but hadn't thought them through.  I'm sold.
BTW, @mattdailis suggested a similar path. . . he probably even said "instrumentation API."  But, I didn't look into it enough because I thought it was for sampling performance.  I didn't realize it was this general.  Matt, feel free to tell me, "told you so!"   : - )

authors: Mythicaeda Twisol bradNASA

repo: NASA-AMMOS/aerie


text:
Create unit tests for extract_tiles.py

There aren't any unit tests for extract_tiles.py.  These will be very useful for the Python2 to 3 update.

Committed README, custom.conf, extract_tiles_test.conf, and test_extract_tiles.py to the 'develop' branch, prior to the METplus-2.2 version

The README for running this test has revealed that the correct version of the extract_tiles_test.conf file was not committed.  Additionally, re-running this test "from scratch" has uncovered an assumption that the TcPairs wrapper has already been run.  Instead, create a fixture for the tests where TcPairs is always run prior to running the ExtractTiles wrapper.

Don't forget to add to the run_pytests.sh script so that this test can be run with the other existing tests when this script is invoked.

Created entry in run_pytests for extract tiles wrapper tests.  Tests take longer because tc_pairs output needed and created prior to running test.

Hi Minna, could we stage some tc_pairs output for use in testing so we
don't have to run it for these tests?
On Fri, Oct 25, 2019 at 12:57 PM bikegeek notifications@github.com wrote:

Created entry in run_pytests for extract tiles wrapper tests. Tests take
longer because tc_pairs output needed and created prior to running test.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/NCAR/METplus/issues/234?email_source=notifications&email_token=AFSSZNZDTJXT7UP644NIGPDQQM6TBA5CNFSM4HS6YKK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECJIELI#issuecomment-546472493,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFSSZN65A7BPT2WTRQDFZ4DQQM6TBANCNFSM4HS6YKKQ
.

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
My working day may not be your working day. Please do not feel obliged to
reply to this email outside of your normal working hours.

Only if we do this just before calling these tests.  I tried using "stale"
tc_pairs output and the extract tiles wrapper is looking for certain temp
files because it's calling tc_stat. I tried so many ways of doing this
within the test itself without success.  We can do this by calling the
run_precondition.py prior to calling run_pytests.sh.  I thought about that,
but then if someone forgets to run that, the tests will fail.
-M
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401
On Fri, Oct 25, 2019 at 1:05 PM George McCabe notifications@github.com
wrote:

Hi Minna, could we stage some tc_pairs output for use in testing so we
don't have to run it for these tests?
On Fri, Oct 25, 2019 at 12:57 PM bikegeek notifications@github.com
wrote:

Created entry in run_pytests for extract tiles wrapper tests. Tests take
longer because tc_pairs output needed and created prior to running test.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<
https://github.com/NCAR/METplus/issues/234?email_source=notifications&email_token=AFSSZNZDTJXT7UP644NIGPDQQM6TBA5CNFSM4HS6YKK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECJIELI#issuecomment-546472493
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AFSSZN65A7BPT2WTRQDFZ4DQQM6TBANCNFSM4HS6YKKQ
.

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
My working day may not be your working day. Please do not feel obliged to
reply to this email outside of your normal working hours.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/NCAR/METplus/issues/234?email_source=notifications&email_token=AA4UJHXUGETNBQ6RRY5L6W3QQM7OXA5CNFSM4HS6YKK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECJIZAQ#issuecomment-546475138,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA4UJHUTBPYUSYJBE33K3RTQQM7OXANCNFSM4HS6YKKQ
.

Could you elaborate on the temp files? Why do we create them and rely on
the temp file name from previous execution?
On Fri, Oct 25, 2019 at 1:42 PM bikegeek notifications@github.com wrote:

Only if we do this just before calling these tests. I tried using "stale"
tc_pairs output and the extract tiles wrapper is looking for certain temp
files because it's calling tc_stat. I tried so many ways of doing this
within the test itself without success. We can do this by calling the
run_precondition.py prior to calling run_pytests.sh. I thought about that,
but then if someone forgets to run that, the tests will fail.
-M
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax: 303-497-8401
On Fri, Oct 25, 2019 at 1:05 PM George McCabe notifications@github.com
wrote:

Hi Minna, could we stage some tc_pairs output for use in testing so we
don't have to run it for these tests?
On Fri, Oct 25, 2019 at 12:57 PM bikegeek notifications@github.com
wrote:

Created entry in run_pytests for extract tiles wrapper tests. Tests
take
longer because tc_pairs output needed and created prior to running
test.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<

https://github.com/NCAR/METplus/issues/234?email_source=notifications&email_token=AFSSZNZDTJXT7UP644NIGPDQQM6TBA5CNFSM4HS6YKK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECJIELI#issuecomment-546472493

,
or unsubscribe
<

https://github.com/notifications/unsubscribe-auth/AFSSZN65A7BPT2WTRQDFZ4DQQM6TBANCNFSM4HS6YKKQ

.

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
My working day may not be your working day. Please do not feel obliged to
reply to this email outside of your normal working hours.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<
https://github.com/NCAR/METplus/issues/234?email_source=notifications&email_token=AA4UJHXUGETNBQ6RRY5L6W3QQM7OXA5CNFSM4HS6YKK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECJIZAQ#issuecomment-546475138
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AA4UJHUTBPYUSYJBE33K3RTQQM7OXANCNFSM4HS6YKKQ
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/NCAR/METplus/issues/234?email_source=notifications&email_token=AFSSZN6NP7Y6LPLQOZ5P7R3QQNDZ3A5CNFSM4HS6YKK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECJLV4I#issuecomment-546487025,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFSSZNY72QNIVMHGHSHUMFLQQNDZ3ANCNFSM4HS6YKKQ
.

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
My working day may not be your working day. Please do not feel obliged to
reply to this email outside of your normal working hours.

I know that the tc_stat wrapper throws an error when there are some /tmp
files missing.  These contain a list of analysis and forecast files- I
don't remember why these are needed, but this is how things work.  When we
have more time and money, we can refactor the feature relative use cases to
fit in with the rest of the wrappers.

Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401
On Fri, Oct 25, 2019 at 2:11 PM George McCabe notifications@github.com
wrote:

Could you elaborate on the temp files? Why do we create them and rely on
the temp file name from previous execution?
On Fri, Oct 25, 2019 at 1:42 PM bikegeek notifications@github.com wrote:

Only if we do this just before calling these tests. I tried using "stale"
tc_pairs output and the extract tiles wrapper is looking for certain temp
files because it's calling tc_stat. I tried so many ways of doing this
within the test itself without success. We can do this by calling the
run_precondition.py prior to calling run_pytests.sh. I thought about
that,
but then if someone forgets to run that, the tests will fail.
-M
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax: 303-497-8401
On Fri, Oct 25, 2019 at 1:05 PM George McCabe notifications@github.com
wrote:

Hi Minna, could we stage some tc_pairs output for use in testing so we
don't have to run it for these tests?
On Fri, Oct 25, 2019 at 12:57 PM bikegeek notifications@github.com
wrote:

Created entry in run_pytests for extract tiles wrapper tests. Tests
take
longer because tc_pairs output needed and created prior to running
test.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<

https://github.com/NCAR/METplus/issues/234?email_source=notifications&email_token=AFSSZNZDTJXT7UP644NIGPDQQM6TBA5CNFSM4HS6YKK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECJIELI#issuecomment-546472493

,
or unsubscribe
<

https://github.com/notifications/unsubscribe-auth/AFSSZN65A7BPT2WTRQDFZ4DQQM6TBANCNFSM4HS6YKKQ

.

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<

https://github.com/NCAR/METplus/issues/234?email_source=notifications&email_token=AA4UJHXUGETNBQ6RRY5L6W3QQM7OXA5CNFSM4HS6YKK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECJIZAQ#issuecomment-546475138

,
or unsubscribe
<

https://github.com/notifications/unsubscribe-auth/AA4UJHUTBPYUSYJBE33K3RTQQM7OXANCNFSM4HS6YKKQ

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<
https://github.com/NCAR/METplus/issues/234?email_source=notifications&email_token=AFSSZN6NP7Y6LPLQOZ5P7R3QQNDZ3A5CNFSM4HS6YKK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECJLV4I#issuecomment-546487025
,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AFSSZNY72QNIVMHGHSHUMFLQQNDZ3ANCNFSM4HS6YKKQ
.

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
My working day may not be your working day. Please do not feel obliged to
reply to this email outside of your normal working hours.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/NCAR/METplus/issues/234?email_source=notifications&email_token=AA4UJHSO25LFHSXB7RR6OEDQQNHGHA5CNFSM4HS6YKK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECJNZ7I#issuecomment-546495741,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA4UJHSPHXRH3VSW2NQKUMTQQNHGHANCNFSM4HS6YKKQ
.

Update, John and I looked at the extract tiles wrapper and we do indeed need to keep the code as it is, where the temporary file has the current process ID appended to it.  This is done to prevent concurrent runs clobbering the output.  The information in the temporary file is used to create the final output names.  The logic employed in the wrapper is based on the original ksh script that performed the extraction of tiles.

source: gharchive/issue

authors: bikegeek georgemccabe

repo: NCAR/METplus

license_type: permissive

license_source: github-api


text:
CI is failing: use of deprecated scipy functionality

It looks like we need to update NCL_vector_5.py.
See here: https://github.com/NCAR/geocat-examples/actions/runs/9239185399/job/25418034302

This failure is because interp2d was recently removed from SciPy in v1.14.0.  There's a transition guide available from SciPy to help adapt codes.
Adapting the code shouldn't be too much of a lift, but is more involved than swapping out a function or two.
Also, in looking into the impacted example NCL_vector_5.py more deeply, it seems we're really using this interpolation function as a proxy for more sophisticated regridding in order to get the data on a grid that can be used by matplotlib's streamplot.  There are two interrelated issues here as well.

I think ideally we wouldn't need to provide data on a regular grid.  There's a related issue open over on matplotlib, but it's gone stale.  In the geosciences in particular this is a feature of interest.  A brief skim of that issue also leads me to think that there may not be checks for this in the matplotlib code so there might be some incorrect visualizations out there because of this.  This is beyond the scope of this issue to address, but might be something worth considering.  I'm also curious if it's come up with UXarray at all.
We should probably do something a little different for the regridding rather than using interp2d or a replacement.  I'll explore this a bit more and see what the original example does, if anything.

repo: NCAR/geocat-examples


text:
Add notification banner message when there is a new version of ANET

When a new version of ANET is published, it is problematic for users that leave their browsers open overnight, they end up with a version out of sync.
Have ANET check for new versions, and when the case, automatically reload the client if the page contains no unsaved data.
Add notification banner message when not possible
related to #3019 and #1164

The primary challenge on this issue is to detect when a new version is published. Version number must, somehow, come from server. There are two possibilities here;

Server pushes version number to client whenever a new version is deployed. Can be implemented in several ways:

Websockets

Advantages:

Clients gets notified about new versions immediately.
Easy to implement on client-side with small overhead.
Can also be used for offline detection (#1164)

Disadvantages:

Requires dedicated changes on server-side.
Might be overkill (unnecessarily resource intensive) just for version updates.

Web Push API

Advantages:

Standard, well recognized API
Clients gets notified about new versions immediately.
Easy to implement on client-side with medium overhead.
Once implemented can be used for other use cases (such as notifying user for new published reports)

Disadvantages:

Requires permission from user on user agent. If user declines, whole thing falls down.
Requires dedicated changes on server-side, probably with medium/large overhead.

Client asks/fetches/pulls current version number from server and compares it to the last version it knows to detect if there is an update.

Adding a dedicated GraphQL API for client version.

Advantages:

Clients gets notified about new versions almost immediately.
Can be implemented side-by-side with #3019
If version field can be added to all server graphql responses whole thing can be implemented with minimal overhead. (TODO: clarify)

Disadvantages:

If version is not send with every response then we need to poll it. Polling has a major trade off:

Short polling intervals results in more immediate new version detection while consuming more resources (i.e. unnecessary requests)
Long polling intervals results in delayed new version detection while consuming less resources.

Utilize Service Workers for update management. example

Advantages:

Standard API, widely used for SPA version updates.
Whole thing can be implemented client side. (TODO: make sure about this)

Disadvantages:
2. A bit difficult to implement.

NOTE: This proposal is not finalized, it is still in progress.

As we are using mainly graphql, I suggest graphql subscriptions.

authors: VassilIordanov ysf-simsoft

repo: NCI-Agency/anet

license: mit

license_source: bigquery


text:
Default Sort on Table

What problem does this feature solve?
Set deafult sorts on table and show values
What does the proposed API look like?
Add on table header
<th nzShowSort nzSortKey="Inventory" nzSortDirection="[asc/desc]" >ColumnaName</th>

https://ng.ant.design/components/table/en#components-table-demo-custom-filter-panel
we already support it, you can check it here.

In the example provided ther's no reference to SORTING

You can do
<th nzShowSort nzSortKey="your_sort_key" nzSort="ascend">            ...           </th>
the available values are 'ascend' | 'descend' | null;

authors: MassimilianoOreto adaczo oluevano vthinkxie

repo: NG-ZORRO/ng-zorro-antd

license: MIT


text:
Not able to dynamically hide the tooltip based on the condition

Reproduction link
https://stackblitz.com/edit/angular-mjbd3a?file=src/app/app.component.ts
Steps to reproduce
In the example click on the VISIBLE button, when the visible is false then it should not show the tooltip and if visible is true, then it show the tooltip.
What is expected?
should show/hide the tooltip dynamically. In previous version, it was working.
What is actually happening?
Current version its not working as expected.

Environment
Info

ng-zorro-antd
9.0.2

Browser
Chrome 81

This PR also helps to solve the problem.

Thank you for fixing this. 👍


text:
[Progress] support successColor

https://github.com/ant-design/ant-design/pull/24655

OK


text:
Added a map

Added a map functionality to better mimic the old looking glass

Congrats and thank you for being the first person to do PR!
Thanks for your work, this looks nice! I'm going to merge this, but may consider replacing the button for generating the map with something else in the future (maybe a separate option in the form).

source: gharchive/pull-request

authors: FHRNet teunvink

license: ISC


text:
支持lucene的ansjAnalyzer分词器好像参数有问题

你说的对，我也发现了这个问题！奇怪这么严重的问题到ES插件那边怎么还能工作……

authors: nfjava xinlmain


text:
Simply assume Windows is Little Endian

Fixing support for using autoconf when used as a submodule in nsd. I've found no occurences of Windows ever running in Big Endian mode, even on PowerPC (nice read), for which support was retired after NT4.0.

I also do not know of an instance where Windows ran in big endian.
Currently, Windows run also on ARM processors, but in little endian mode.

repo: NLnetLabs/simdzone


text:
System.InvalidOperationException: Collection was modified

Hi,
We started noticing this kind of exception in our .NET Core project:
{
  "ClassName": "NLog.NLogRuntimeException",
  "Message": "Exception occurred in NLog",
  "Data": {
    "NLog.ExceptionLoggedToInternalLogger": true
  },
  "InnerException": {
    "ClassName": "System.InvalidOperationException",
    "Message": "Collection was modified; enumeration operation may not execute.",
    "Data": {
      "NLog.ExceptionLoggedToInternalLogger": true
    },
    "InnerException": null,
    "HelpURL": null,
    "StackTraceString": "   at System.Collections.Generic.List`1.Enumerator.MoveNextRare()\r\n   at System.Collections.Generic.List`1.Enumerator.MoveNext()\r\n   at NLog.Targets.ElasticSearch.ElasticSearchTarget.FormPayload(ICollection`1 logEvents)\r\n   at NLog.Targets.ElasticSearch.ElasticSearchTarget.SendBatch(ICollection`1 logEvents)",
    "RemoteStackTraceString": null,
    "RemoteStackIndex": 0,
    "ExceptionMethod": null,
    "HResult": -2146233079,
    "Source": "System.Private.CoreLib",
    "WatsonBuckets": null
  },
  "HelpURL": null,
  "StackTraceString": "   at NLog.LoggerImpl.<>c__DisplayClass1_0.<Write>b__1(Exception ex)\r\n   at NLog.Internal.SingleCallContinuation.Function(Exception exception)\r\n   at NLog.Targets.ElasticSearch.ElasticSearchTarget.SendBatch(ICollection`1 logEvents)\r\n   at NLog.Targets.ElasticSearch.ElasticSearchTarget.Write(AsyncLogEventInfo logEvent)\r\n   at NLog.Targets.Target.WriteAsyncThreadSafe(AsyncLogEventInfo logEvent)\r\n   at NLog.Targets.Target.WriteAsyncLogEvent(AsyncLogEventInfo logEvent)\r\n   at NLog.LoggerImpl.WriteToTargetWithFilterChain(Target target, FilterResult result, LogEventInfo logEvent, AsyncContinuation onException)\r\n   at NLog.LoggerImpl.Write(Type loggerType, TargetWithFilterChain targetsForLevel, LogEventInfo logEvent, LogFactory factory)\r\n   at NLog.Logger.Log(Type wrapperType, LogEventInfo logEvent)\r\n   at NLog.Extensions.Logging.NLogLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)\r\n   at Microsoft.Extensions.Logging.Logger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)",
  "RemoteStackTraceString": null,
  "RemoteStackIndex": 0,
  "ExceptionMethod": null,
  "HResult": -2146233088,
  "Source": "NLog",
  "WatsonBuckets": null
}

Which I am not sure if it is a bug in the NLog internal library or something on our side ... but I have seen some other tickets open, related to this exception, in the past (4-5 years ago, and the tickets are closed, so I doubt we have the same issue as those).
We are using the following packages:
<PackageReference Include="NLog" Version="4.6.6" />     <PackageReference Include="NLog.Targets.ElasticSearch" Version="5.1.0" /> <PackageReference Include="NLog.Web.AspNetCore" Version="4.8.3" /> <PackageReference Include="NLog.Extensions.Logging" Version="1.5.2" />
In a .NET Core 2.2 project, which is hosted in an Azure Function.
I'd like to mention one thing; that there is some parallel threading going on in our solution (i.e. I am logging the execution of our function on a parallel thread)... I am not sure if that could cause the issue (because this type of exception is something I've seen in the past where some sort of collection was not thread safe). However I would assume the ILogger framework provided in .NET Core is thread safe
Thoughts ?
Vlad

Readable stacktraces:
 at NLog.LoggerImpl.<>c__DisplayClass1_0.<Write>b__1(Exception ex)
   at NLog.Internal.SingleCallContinuation.Function(Exception exception)
   at NLog.Targets.ElasticSearch.ElasticSearchTarget.SendBatch(ICollection`1 logEvents)
   at NLog.Targets.ElasticSearch.ElasticSearchTarget.Write(AsyncLogEventInfo logEvent)
   at NLog.Targets.Target.WriteAsyncThreadSafe(AsyncLogEventInfo logEvent)
   at NLog.Targets.Target.WriteAsyncLogEvent(AsyncLogEventInfo logEvent)
   at NLog.LoggerImpl.WriteToTargetWithFilterChain(Target target, FilterResult result, LogEventInfo logEvent, AsyncContinuation onException)
   at NLog.LoggerImpl.Write(Type loggerType, TargetWithFilterChain targetsForLevel, LogEventInfo logEvent, LogFactory factory)
   at NLog.Logger.Log(Type wrapperType, LogEventInfo logEvent)
   at NLog.Extensions.Logging.NLogLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at Microsoft.Extensions.Logging.Logger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)",

Inner:

 at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
   at System.Collections.Generic.List`1.Enumerator.MoveNext()
   at NLog.Targets.ElasticSearch.ElasticSearchTarget.FormPayload(ICollection`1 logEvents)
   at NLog.Targets.ElasticSearch.ElasticSearchTarget.SendBatch(ICollection`1 logEvents)",

If looks like the issue in in  NLog.Targets.ElasticSearch.ElasticSearchTarget. It guess that's this repo: https://github.com/markmcdowell/NLog.Targets.ElasticSearch

Bug has been found: https://github.com/markmcdowell/NLog.Targets.ElasticSearch/issues/113

authors: 304NotModified andronachev snakefoot

repo: NLog/NLog


text:
Error Facing while connecting to web service

I want to log my Exceptions from my .Net Application to the Airtable using the NLog
After configuring the nlog.config file. getting the following error in the Nlog internal logs:
Error WebServiceTarget(Name=ws): Error sending request Exception: System.Net.WebException: The remote server returned an error: (422) Unprocessable Entity.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at NLog.Targets.WebServiceTarget.<>c__DisplayClass67_0.<CreateSendContinuation>b__1(IAsyncResult r)

Here is my nlog.config file
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"
      throwConfigExceptions="true"
      internalLogLevel="info"
      internalLogFile="c:\temp\internal-nlog-AspNetCore3.txt" >
   <extensions>
    <add assembly="NLog.Targets.Http" />
  </extensions>
  <targets>
     <target type='WebService'
            name='ws'
            url='https://api.airtable.com/v0/BaseID/Log?api_key=APIKEY'
            protocol="JsonPost"
            encoding="UTF-8">
            <parameter name="fields">
               <layout xsi:type="JsonLayout" includeAllProperties="true">
                    <attribute name="Description" layout="teswt" />
               </layout>
            </parameter>
        </target>
    </targets>
    <rules>
      <logger name='*' writeTo='ws'></logger>
    </rules>
</nlog>

AirTable wants a JSON like:
{
    "fields": {
        "Description": "test1"
    }
}

Can anyone let me know that the above targets will provide the above JSON or not?
Also, is there a way to log the request sent to the webservice like the JSON Payload, the URl etc.

Maybe you can do this:
     <target type='WebService'
            name='ws'
            url='https://api.airtable.com/v0/BaseID/Log?api_key=APIKEY'
            protocol="JsonPost"
            encoding="UTF-8">
       <parameter name=""><layout xsi:type="JsonLayout">
          <attribute name="fields" encode="false">
               <layout xsi:type="JsonLayout" includeAllProperties="true">
                   <attribute name="Description" layout="test1" />
               </layout>
          </attribute>
       </layout></parameter>
     </target>

Did it answer your question?

Thanks for the help.
@snakefoot comment leaded me to the desired result

authors: anilk-enest snakefoot


text:
combine "Connecting" and "Logging In" pages into a single document

Most of the information in "Connecting" is for Gaea.  Anthing useful that isn't already in the Gaea user guide should be placed there.  Anything not in "Logging In" should be moved to that page.  Then "Connecting" should be removed.

addressed as part of PR #189

authors: chanwilson underwoo


text:
Doorbell event

Hey,
I would love to create a nodered event when the G4 doorbell is pressed, I get a websocket trigger. I tried using your plugin but the websocket doesn't work.
To help you on your way, I figured the following out:
There's a websocket for which shows the doorbell popup when opening a protect page. The url is wss://192.168.1.1/proxy/protect/ws/updates?lastUpdateId=2cdea4f6-cb22-4dfe-8bc1-2c866f3516bf. It sends binairy files which are described here: https://github.com/hjdhjd/homebridge-unifi-protect and here https://github.com/hjdhjd/homebridge-unifi-protect/blob/master/src/protect-api-updates.ts
The lastUpdateId you can get using https://192.168.1.1/proxy/protect/api/bootstrap (in JSON lastUpdateId field).
Would love to get that websocket working!
Thanks

@crxporter Something new :)
I am going to look at it soon.

This is actually the main goal why I wanted to make this plugin. It's essentially what started the project.
All other information can be gathered by HTTP requests.
This request is essentially to monitor the live updates from UniFi Protect including but not limited to:

motion alerts
doorbell alerts
smart detections from G4 cams
night IR light on/off statu
likely the new sensor data...

So - yes. This is a huge very important goal of the project.
Shaquu - the incoming data is going to be in binary form. There are some bits which will tell where the JSON starts and ends, it will need to be loaded as a buffer then parsed.
This is a dev test of what we are looking for. I haven't actually tried it but potentially the output on this script will be the exact JSON payloads we want to pump into nodered.

Good to here. I did read the readme, but thought maybe the info I provided wasn´t something you were aware off.

@schwickster if it is related to UniFi Protect API then you should try 0.3.0 #23

I've got a bell. Confirmed this works - readme has been updated with some instructions as well.
Please provide any feedback or comments you have! Thanks!

Yup I finally tested it for a few days. Getting a lot of error 1002 and 4000 though

authors: Shaquu crxporter schwickster

repo: NRCHKB/node-red-contrib-unifi-os


text:
Add new sub-domains with AWS named server (NS)

Request from Naeem.

Hello Gabriel,
I am working with Eric to develop/migrate GSIP (https://geoconnex.ca/gsip/app/index.html) application to AWS and wanted to know if we can have a subdomain for our AWS networking and make an entry in the DNS provider for us:
LDRS.geoconnex.ca
I can provide you the NS records once I get confirmation from you.
Thanks

@denevers

After trying to find how it's being handled with EnomCentral, Naeem and me agreed to ask for support from their helpdesk.

This github repo is just for the code, not the infrastructure. I suggest you move this concern to internal gitlab.
Closing this ticket

authors: denevers ghuotvez

repo: NRCan/gsip


text:
debug steamturbine and require input for Boiler

Require fuel_cost_per_mmbtu as an input for Boiler tech
Correct how steam turbine defaults are determined

I just realized that this PR was created back in March when we were last working on adding these techs into REopt_API. It should have some relevant changes.


text:
Consider changing time formatting in LiveTable.

@cmazzoli: "Print start time at the top and then (rounded) elapsed seconds in the table, conserving horizontal characters."

Closed by #154

authors: ambarb danielballan

repo: NSLS-II/bluesky


text:
FIX: plan pattern args motor repr

Yeah, at least the second time around with these motor reprs; this time in the plan pattern args.
Added bson encode test in test_simple_api. This should not add a bson dependency as it is pulled in through metadatastore (which I assume is in nslsii_dev_configuration)

[Current coverage][cc-pull] is 81.59%

Merging [#470][cc-pull] into [master][cc-base-branch] will increase coverage by <.01%

@@             master       #470   diff @@
==========================================
  Files            18         18
  Lines          3741       3747     +6
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           3051       3057     +6
  Misses          690        690
  Partials          0          0

Powered by Codecov. Last updated by [65448c0...416f387][cc-compare]
[cc-base-branch]: https://codecov.io/gh/NSLS-II/bluesky/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/NSLS-II/bluesky/compare/65448c035da9610ce9c89dfa3c9ebf746038a151...416f387a15de3ddac203df5ff57ff14d559cc972
[cc-pull]: https://codecov.io/gh/NSLS-II/bluesky/pull/470?src=pr

Why not set the plan_pattern_args as part of the ChainMap where all the other md is set? It's a minor readability/consistency improvement, but more importantly it makes that key overridable if the caller knows something that the hard-coded function does not.

I think doing it this way is better, the caller can not know more about this than we know locally (you should not be able to lie about which plan_pattern_args where used as this is the final authority on what was used).

authors: codecov-io danielballan klauer tacaswell


text:
Speaker order is randomised while loading - why?

I was wondering whether this is intentional or not, but it seems the utterances from the file_list are shuffled before loading. I'm talking about these lines. It makes selecting an audio file for style transfer difficult, so I was wondering if there is a reason for this.

Ok, that was actually a stupid question because you are also using the loader for training. My bad. But maybe it would make more sense to make it optional with a flag during style transfer?

repo: NVIDIA/mellotron


text:
GraphCast loss function different from original DeepMind version

I noticed that the graphcast loss function in this repository differs from what was used in the original graphcast paper, since the implemented here does not weight loss based on pressure level or type of variable. Just wondering if there was a reason for this or if there are plans to implement the level/variable weighting in the graphcast loss function in future updates?

Hi @cr-metdev , thanks for posting your question. We started with a simplified version of the loss function, but now in a recent PR, the implementation has improved to include other loss weightings based on pressure level or type of variable: https://github.com/NVIDIA/modulus/pull/554

Closing as complete

authors: cr-metdev mnabian

repo: NVIDIA/modulus


text:
[BUG] Parquet Chunked reader causing performance overhead

Describe the bug
I have a customer job where the parquet chunked reader is causing the stage to go from 2.1 minutes to 3 minutes.
It looks like with the chunked reader we have about 3 times the number of batches and the shuffle read and/or coalescebatches is taking much longer.
Needs more investigation.

I was wrong on the number of batches, was looking at the wrong stage, close to same number of batches but consistently faster without chunked reader. going to try to get trace

So there were 2 jobs here. The one I'm concentrating on did have 3x the number of batches. Presumably the estimation of the maxBytes and readmax bytes was off and the chunked reader was closer to what it was really set at 128m per batch size.
The issue here is actually with the multithreaded shuffle reader and how the new limiter is working. It looks like the limiter and code were causing it to wait to put more things in the queue even though it has data ready. This got worse with the chunked reader because it had more batches.

I'm not seeing this now when run through the entire application, going to lower priority.

closing since not able to reproduce can reopen if we see it again

repo: NVIDIA/spark-rapids


text:
Fix NPE in plugin shutdown

Signed-off-by: Andy Grove andygrove73@gmail.com
I was trying to run the build on a desktop with an RTX 3080 and hit a CUDA memory issue, which resulted in this NPE which is annoying and easily avoided.

build

build


text:
Can't load pretrained models in inference.ipynb

I'm just trying to use the published pre-trained models with the default ipynb file and receiving the following errors:

RuntimeError                              Traceback (most recent call last)
 in
1 checkpoint_path = "tacotron2_statedict.pt"
2 model = load_model(hparams)
----> 3 model.load_state_dict(torch.load(checkpoint_path)['state_dict'])
4 _ = model.cuda().eval().half()
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
719         if len(error_msgs) > 0:
720             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 721                                self.class.name, "\n\t".join(error_msgs)))
722
723     def parameters(self):
RuntimeError: Error(s) in loading state_dict for Tacotron2:
Unexpected key(s) in state_dict: "encoder.convolutions.0.1.num_batches_tracked", "encoder.convolutions.1.1.num_batches_tracked", "encoder.convolutions.2.1.num_batches_tracked", "postnet.convolutions.0.1.num_batches_tracked", "postnet.convolutions.1.1.num_batches_tracked", "postnet.convolutions.2.1.num_batches_tracked", "postnet.convolutions.3.1.num_batches_tracked", "postnet.convolutions.4.1.num_batches_tracked".
And further when I'm trying to load yours waveglow model:

NotSupportedError                         Traceback (most recent call last)
 in
1 waveglow_path = 'waveglow/waveglow_256channels.pt'
----> 2 waveglow = torch.load(waveglow_path)['model']
3
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/serialization.py in load(f, map_location, pickle_module)
301         f = open(f, 'rb')
302     try:
--> 303         return _load(f, map_location, pickle_module)
304     finally:
305         if new_fd:
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/serialization.py in _load(f, map_location, pickle_module)
467     unpickler = pickle_module.Unpickler(f)
468     unpickler.persistent_load = persistent_load
--> 469     result = unpickler.load()
470
471     deserialized_storage_keys = pickle_module.load(f)
~/tacotron2/waveglow/glow.py in
31
32
---> 33 @torch.jit.script
34 def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels):
35     n_channels_int = n_channels[0]
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/init.py in script(fn)
535
536 def script(fn):
--> 537     graph = _script_graph(fn, frame_id=3)
538     return torch._C.GraphExecutor(graph, True)
539
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/init.py in _script_graph(fn, frame_id)
530 def _script_graph(fn, frame_id=2):
531     rcb = createResolutionCallback(frame_id)
--> 532     ast = get_jit_ast(fn)
533     return _jit_script_compile(ast, rcb)
534
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/frontend.py in get_jit_ast(fn)
125     if len(py_ast.body) != 1 or not isinstance(py_ast.body[0], ast.FunctionDef):
126         raise RuntimeError("expected a single top-level function")
--> 127     return build_def(SourceRangeFactory(source), py_ast.body[0])
128
129
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/frontend.py in build_def(ctx, py_def)
144     return Def(Ident(r, py_def.name),
145                build_param_list(ctx, py_def.args),
--> 146                [build_stmt(ctx, stmt) for stmt in body])
147
148
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/frontend.py in (.0)
144     return Def(Ident(r, py_def.name),
145                build_param_list(ctx, py_def.args),
--> 146                [build_stmt(ctx, stmt) for stmt in body])
147
148
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/frontend.py in call(self, ctx, node)
133         if method is None:
134             raise UnsupportedNodeError(ctx, node)
--> 135         return method(ctx, node)
136
137
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/frontend.py in build_Assign(ctx, stmt)
192     @staticmethod
193     def build_Assign(ctx, stmt):
--> 194         rhs = build_expr(ctx, stmt.value)
195         if len(stmt.targets) > 1:
196             start_point = ctx.make_range(stmt.lineno, stmt.col_offset, stmt.col_offset + 1)
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/frontend.py in call(self, ctx, node)
133         if method is None:
134             raise UnsupportedNodeError(ctx, node)
--> 135         return method(ctx, node)
136
137
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/frontend.py in build_Call(ctx, expr)
297     def build_Call(ctx, expr):
298         func = build_expr(ctx, expr.func)
--> 299         args = [build_expr(ctx, py_arg) for py_arg in expr.args]
300         if hasattr(expr, 'starargs') and expr.starargs:
301             stararg_expr = build_expr(ctx, expr.starargs)
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/frontend.py in (.0)
297     def build_Call(ctx, expr):
298         func = build_expr(ctx, expr.func)
--> 299         args = [build_expr(ctx, py_arg) for py_arg in expr.args]
300         if hasattr(expr, 'starargs') and expr.starargs:
301             stararg_expr = build_expr(ctx, expr.starargs)
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/frontend.py in call(self, ctx, node)
133         if method is None:
134             raise UnsupportedNodeError(ctx, node)
--> 135         return method(ctx, node)
136
137
~/anaconda2/envs/myenv/lib/python3.6/site-packages/torch/jit/frontend.py in build_Subscript(ctx, expr)
404             return Slice(base, lower, upper)
405         elif sub_type is ast.ExtSlice:
--> 406             raise NotSupportedError(base.range(), "slicing multiple dimensions at the same time isn't supported yet")
407         else:  # Ellipsis (can only happen in Python 2)
408             raise NotSupportedError(base.range(), "ellipsis is not supported")
NotSupportedError: slicing multiple dimensions at the same time isn't supported yet
@torch.jit.script
def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels):
n_channels_int = n_channels[0]
in_act = input_a+input_b
t_act = torch.tanh(in_act[:, :n_channels_int, :])
~~~~~~ <--- HERE
s_act = torch.sigmoid(in_act[:, n_channels_int:, :])
acts = t_act * s_act
return acts
My system is OpenSuse 43.2 and I'm using Python 3.6 on anaconda.

Looks like you have a pytorch version mismatch.

What do you mean by that? My version is torch==1.0.1.post2 with
torchvision==0.2.2. So it's version you said to install in setup instruction.


text:
Mainline Shield Tablet(st8) support - Weird framebuffer issue.

Visualization:
https://imgur.com/a/KWjF0
https://youtu.be/-yHrsPXEJ2U
Kernel seems to boot fine at the current tip of linux next: (f7ac28a6971b43a2ee8bb47c0ef931b38f7888cf). However it seems like either tegrafb, nvidiafb, or nouveaufb tries to takeover unsuccessfully as seen in the photo/video.
When using @Gnurou's st8 branch it boots farther, gets into userspace and runs init. Then kernel-panics after init fails to mount the rootfs.(I have busybox in the boot image; shouldn't the kernel just drop me down to a minimal shell to mount root manually if it cannot switch to the "real" root?)
Having access to a TTY level or RS232 level interface would help!
Few questions to get the full boot-log:

Does the shield tablet's/tegra's echi controller offer a debug port? (The specific tegra chipset for the shield tablet is the TD575D-A1)
Is it possible nvidia could make public if the JTAG/RS-232 interface pins are routed out to an unpopulated connecter on the PCB?

(Fun facts on the PCB there is an unpopulated Mini-PCIE connecter and unpopulated spots for another 2GB of ram.)
The kernel config used:
https://raw.githubusercontent.com/Gnurou/linux/st8/arch/arm/configs/tegra_defconfig
This is the current device tree that uses simplefb: (decompiled after it was compiled in the kernel tree)
https://gist.github.com/winsock/680938e51501c31a5335
Warnings from dtc when decompiling the binary blob

Warning (ranges_format): /ahub@70300000 has empty "ranges" property but its #address-cells (1) differs from / (2)
Warning (ranges_format): /ahub@70300000 has empty "ranges" property but its #size-cells (1) differs from / (2)

Pinging @Gnurou - Taking you up on your offer!

@Gnurou, I think I have it it userspace now. Problem is lack of an OTG cable since it didn't auto connect to my wifi(I set it up while chrooted in it with sshd). Will let you know of any progress on using nouveaufb.

@Gnurou Did you test/tried to use a keyboard or actually used the tablet when you booted to userspace? I cannot seem to get OTG working. Enabled the OTG support in the kernel and every USB Gadget type. Tegra EHCI is compiled in. I see nothing in the log about usb controller init though.

Some progress, I ported the rm31080a touchscreen driver from the 3.10 tegra tree. Bad news still cannot figure out how to enable OTG. I think the issue is that the regulator isn't switching to providing power when a OTG cable is connected and/or a misconfigured device tree. I have been cobbling it together from string dumps of binary blobs and kernel boot logs of android to see what devices are enabled and at what addresses. Here is the messy DTS though if you want to take a look:
https://github.com/winsock/linux-next-st8/blob/st8/arch/arm/boot/dts/tegra124-st8.dts

Hey, sorry for the delay - I am currently traveling.
The picture and video you post seem to indicate that a display clock is turned down. This would happen if a required clock is not referenced in the device tree. You can probably mitigate this by adding the clk_ignore_unused option to your kernel command-line.
I suppose that the video you posted is still using simplefb, as in my tree? If you want to switch to a real driver, you will probably have to remove simplefb first. On ST8 the display driver to use is tegradrm - Nouveau is only used for the GPU, not for display. For a first time I'd suggest to keep simplefb until you can at least get decent input.
OTG is something I am less familiar with. I am not sure about the state of OTG support in the Tegra USB driver. You may want to have a look at the ST8 Android tree to see whether everything is in place. I never used USB in my tree.
I agree that a serial console would be very useful. Since you mention the PCB I assume that you have opened your ST8? If so I can have a look at the schematics to see if the pins are visible. Alternatively, I suspect that you can twiddle the pinmux to route the serial lines over the SD card pins. Then if you can get your hands on a micro-SD sniffer, you should be able to have a serial console (the voltages will probably need to be adjusted somehow - I am not a hardware guy so don't know much about the details).
I will try to have a closer look at your DT in case I can see what is missing for OTG - since I am on the move, it may take a few days, apologies for that! :)

A few possibly useful links for the OTG issue:
https://devtalk.nvidia.com/default/topic/872807/usb-otg-functionality-in-jetson-tk1/
https://devtalk.nvidia.com/default/topic/897104/jetson-tk1-usb-otg-client-mode/
These concern L4T though. Mainline is supposed to be less feature-complete, so the tegra-otg driver may not even exist there. @thierryreding , do can you comment on the state of OTG on T124 mainline and whether there is an easy way to put a controller back in host mode?

Also what if you change the dr_mode property of the USB phy to "host"? Aren't use able to plug a USB device and use it?

@Gnurou I have two Shields, one taken apart and in "bare" pcb form, and one that is still together. I found that you in fact can put UART over the sd-card interface. It is actually in the st8 3.10 code:
http://nv-tegra.nvidia.com/gitweb/?p=linux-3.10.git;a=blob;f=arch/arm/mach-tegra/board-ardbeg.c;h=6aeba0fdc4ce0f8dbbefda5677535e59761a040a;hb=rel-st8-l-r7-shieldtablet8#l246
You are correct the tegra-otg driver was not in the mainline code, I'll try to port it. If that doesn't make a difference I'll try to set the port to host only mode but the fact that there is no 5v out on the usb port means that I would have to splice in power to make any device work. I'm guessing that's part of what the OTG driver does, signaling the usb regulator to switch to sourcing power instead of sinking it.

Is it possible to get a document on all(Or most of) of the ST8 hardware specifications if such a document exists? I think part of my issue is that as it stands my device tree is not functioning too well and possibly re-writting one from scratch without influence from the android 3.10 tree might help remove some bugs.

Some clarification here from "nvidia,tegra124-car.txt" would be nice also:
- clocks : Should contain phandle and clock specifiers for two clocks:
  the 32 KHz "32k_in", and the board-specific oscillator "osc".

None of the existing device trees in mainline for tegra have any reference to the board-specific oscillator "osc." Is it needed? I also cannot find it even in the Android device trees.

I actually cannot get simplefb working again. I reset my branch to the tip of linux-next and started the device tree from scratch and got rid of all of the warnings that I had before. I see nothing preventing simplefb from starting, however all I get now is what I showed in the first post. I actually really doubt it is a pixel clock issue since I set it implicitly to what I found in the output of the console during android boot.
<7>[    0.140326] disp1 pclk=155774400
<7>[    0.140337] disp2 pclk=297000000
I am assuming DC 1 corresponds to the DSI interface however. Having a skim over the TRM of the K1 was useful in this attempt.
@Gnurou Do you know if the pinout/pinmux of Jetson is similar to the Shield Tablet? (For the devices it does trace out?) Since I do have that document being a registered embedded developer at nvidia.com

Does the Tegra K1 Datasheet apply to the TD575D? There is no mention of that particular chipset.

ST8 layout documents are internal, but I can probably share specific details. That will have to wait until I get back to the office on Jan. 5th. Same thing for trying your tree/code on an actual device.
When the ST8 is powered on, the display is initialized and remains in that state when the kernel takes over. This is that the simple-framebuffer driver takes advantage of (it just writes pixel data into the address of the linear framebuffer used by the bootloader, without any other knowledge of the hardware). If your display breaks, this means the kernel is touching a clock or regulator that affects the display. To avoid this I'd suggest starting with a really minimal DT (and the clk_ignore_unused option) with very few regulators declared. Unless a regulator is declared, the kermel should not touch it, but it tends to switch as many things off as possible as it knows about them. So start with a minimal DT, and add things little by little.
In the past I have also debugged such issues by introducing 1 or 2 seconds delays in the kernel boot sequence to detect when the display corruption happens and "bisect" the exact step (and eventually exact clock/regulator) where it happens. It's a little bit boring, but it works.
For USB, yeah a Y cable that supplies power from another source should be good to get you started - I did the same to run mainline on the SHIELD portable, and it allowed me to use devices. Actually my work on the SHIELD portable might be useful to you too, even though again it's a bit outdated:
https://github.com/linux-shield/kernel/tree/roth
The Jetson TK1 and ST8 are probably too different to use the Jetson as a reference - but a mininal DT to get simplefb to work, and looking at my roth work to enable USB host (while using the exact information contained in the ST8 Android tree) should get you there. I will also try to be more useful when I get back to the office as I am interested in seeing this working! :)

Okay it was actually my memory definition, just using 0x80000000 and a length of 2GiB doesn't work.
This is interesting though, when I uncommented the i2c device tree stuff, the kernel appeared to "hang" and then after 30-45 seconds later then boot.
Also does simplefb provide a drm1.1 interface? If not then I'm somehow using nvidiafb, as I have also seen nvidiafb_init and [drm] Initialized drm 1.1.0.[numbers]

Obviously I don't expect you to reply until you get back to the office, I just will forget to ask these kind of questions if I don't write them out when I think of them! Interesting effect when you remove the linux,initrd-start/end properties in the chosen node. It still boots and I see console text for a split second but the display tries to modeswitch or handoff to nouveau or something and then the display goes blank with backlight on.

It seems like I cannot get init running from my boot image. The only major difference is instead of blindly copying in all the android tegra124 device trees I just heavily based new device trees off the the android ones. Because of that I don't have tegra124-tn8.dsti, tegra124-soc-shield.dtsi, or tegra124-soc.dtsi as well. I just include the mainline tegra124.dtsi directly.

simplefb does not provide a DRM interface. The message that you're seeing is the DRM subsystem initializing. Also, even if you're seeing messages about nvidiafb_init() being called that doesn't mean the driver is actually used. There is no device that the driver could bind to, so most likely it's simply registering, but not doing anything else.
On Tegra the driver that you want for display is Tegra DRM. You can enable it using the DRM_TEGRA symbol. But like @Gnurou said that is likely going to make things harder for now. The easiest is to stick with simplefb and completely disable the Tegra DRM driver for now. If enabled it will try to take over the display and fail, because it doesn't know the panel, and might screw up the existing configuration while at it so that simplefb won't display things properly anymore.

@Gnurou: There is zero support for USB OTG upstream. I've been meaning to look at this for years now, but it was never important enough. It would be nice to get that working, though, because you could do interesting things like a USB serial port or similar. Or, I guess, an ADB.
It's not trivial to do, though, because I don't think there's a driver for the hardware yet. There's some code in U-Boot that can speak device mode (used for the UMS support) but I don't think the kernel has anything equivalent. Also we do have two devices (in Tegra114 and onwards) that support device mode: USBD and XUSB. USBD might be easier to implement because of the existing U-Boot implementation that could serve as a reference. Well, the downstream implementation could of course also be used. I think ChromeOS now has a device mode driver for XUSB and some infrastructure to dynamically switch between host and device mode, so it might be worth investigating if that gets us anywhere.

@winsock hey, so I'm back and can probably be more actively helpful. Is there anything you need me to test / would like information about?

authors: Gnurou thierryreding winsock

repo: NVIDIA/tegra-nouveau-rootfs


text:
Calculating the jacobian for n inputs and n outputs

Below is a sample code for some jacobian calculation that I am trying to perform. My jacobian is around 36000 X 36000 ( I have 36000 inputs and 36000 outputs).
import warp as wp
import numpy as np
from tqdm import tqdm
wp.init()
@wp.kernel
def calc_residue(R : wp.array(dtype=float),
U : wp.array(dtype=float)):
i = wp.tid()
if i==0:
	wp.atomic_add(R, i, 0.01 * U[i]**2.0)
else:

	wp.atomic_add(R, i, 0.01 * U[i]**2.0 +  + U[i-1] ** 2.0)

R = wp.zeros(36000, dtype=float, requires_grad=True)
U = wp.array(np.random.random(36000), dtype=float, requires_grad=True)
tape = wp.Tape()
with tape:
wp.launch(calc_residue, dim = (36000), inputs=[R, U])
jacobian = np.zeros((36000, 36000))
for output_index in tqdm(range(36000)):
select_index = np.zeros(36000)
select_index[output_index] = 1.0
e = wp.array(select_index, dtype=wp.float32)
# pass input gradients to the output buffer to apply selection
tape.backward(grads={R: e})
q_grad_i = U.grad
jacobian[output_index, :] = q_grad_i.numpy()
tape.zero()
Is there a way to vectorize the for loop for constructing the jacobian? In a more complex case (larger computational graph), it takes me around 5 minutes to assemble the jacobian. Is there a way to cut down this time?

Hi @nithinsomu, if the kernel is fairly simple like in your example it would probably make sense to use an explicit Jacobian construction which would be very fast in this case.
@eric-heiden do you have any other suggestion?

Hi @nithinsomu,
The Jacobian can be computed with just 2 tape.backward() calls because every second input index is independent from each other. See these modifications where I create an output index selection array that skips 2 steps. I printed the true Jacobian for a smaller number of dimensions to figure out the assignment indexing:

np.set_printoptions(precision=3, suppress=True, linewidth=400)

dim = 3600
R = wp.zeros(dim, dtype=float, requires_grad=True)
U = wp.array(np.random.random(dim), dtype=float, requires_grad=True)

tape = wp.Tape()

with tape:
	wp.launch(calc_residue, dim = (dim), inputs=[R, U])
jacobian = np.zeros((dim, dim))

for output_index in tqdm(range(dim)):
	select_index = np.zeros(dim)
	select_index[output_index] = 1.0
	e = wp.array(select_index, dtype=wp.float32)
	# pass input gradients to the output buffer to apply selection
	tape.backward(grads={R: e})
	q_grad_i = U.grad
	jacobian[output_index, :] = q_grad_i.numpy()
	tape.zero()

print(jacobian)
print("\n\n")

new_jacobian = np.zeros((dim, dim))
for output_index in range(2):
    select_index = np.zeros(dim)
    selector = np.arange(output_index, dim, 2)
    select_index[selector] = 1.0
    e = wp.array(select_index, dtype=wp.float32)
    # pass input gradients to the output buffer to apply selection
    tape.backward(grads={R: e})
    q_grad_i = U.grad
    new_jacobian[selector, selector] = q_grad_i.numpy()[selector]
    new_jacobian[(selector+2)%dim, (selector+1)%dim] = q_grad_i.numpy()[(selector+1)%dim]
    tape.zero()

print(new_jacobian)

assert np.allclose(jacobian, new_jacobian)

Got it. Thank you very much for the reply.

authors: eric-heiden mmacklin nithinsomu

repo: NVIDIA/warp