text:
[RFC] [VTA] [TSIM] Enabling Cycle-Accurate Hardware Simulation for VTA

The following RFC proposes a new simulation environment called TSIM that improves software and hardware integration and simulation accuracy compared to functional simulation. One of the goals of this RFC is integrating the hardware development process into the software stack from the beginning, allowing features to be incrementally implemented and evaluated as workloads evolve over time.
Under this environment, the hardware description is the actual specification. This reduces the burden of maintaining consistency between the specification written usually in a higher language such as C/C++ and the actual hardware design described in a language such as Verilog. Moving to TSIM will allow us to have a more fluid hardware-software specification, and invite more contributions to modify different layers of the stack.
Moreover, this integration provides a more accurate performance feedback, i.e. clock cycles, compared to the traditional functional model of a hardware accelerator.
This is because TSIM is based on an open-source hardware simulator called Verilator, which compiles Verilog designs down to C++ classes for cycle-accurate simulation.
Lastly, Verilator is already available in many Linux distributions, i.e. Ubuntu, and OSX via homebrew.
Proposed design
TSIM uses Verilator to integrate VTA designs into TVM and provides flexibility in the hardware language used to implement these designs.
For example, one could use OpenCL, C/C++ or Chisel3 to describe a VTA design that would eventually be compiled down to Verilog, since it is the standard input language for FPGA/ASIC tools.
Additionally, Verilator supports the Direct Programming Interface (DPI), which is part of the Verilog standard and a mechanism to support foreign programming languages.
We leverage these features available in Verilator to interface hardware designs from upper layers in the TVM stack such as drivers, runtime, etc. In fact, we have developed all the glue layers to make this happen, including:

DPI module. Based on the DSO module located at tvm/src/runtime/dso_module.cc, the dpi_module.cc is in charge of loading the shared library libtsim.so that contains the hardware accelerator and the Verilator execution function.
As stated earlier, Verilator is used to compile the hardware accelerator from Verilog to C++.
Additionally, the DPI module provides an API that can be used by drivers to manage the accelerator by writing/reading registers and terminate (exit) the simulation.

Verilator execution function. This function is called tsim.cc and it is used by Verilator to instantiate the accelerator, generate clock and reset signals, and dump simulation waveforms when it is enabled. The tsim.cc also contains function pointers to DPI functions which are implemented in the DPI module dpi_module.cc. This adds greater flexibility because the behavior of DPI functions can be modified by upper layers in the stack.

Hardware DPI modules. Normally, a hardware accelerator interface can be simplified in two main components, one for control and another for data. The control interface is driven by a host CPU, whereas the data interface is connected to either external memories (DRAM) or internal memories in the form of scratchpads or caches.
There are two hardware modules written in Verilog implementing these two interfaces called VTAHostDPI.v and VTAMemDPI.v.
Accelerators implemented in Verilog can use these modules directly but we also provide Chisel3 wrappers BlackBox for accelerators described in this language.

Add-by-one accelerator example. To showcase the interaction between all of these components, we implemented an Add-by-one accelerator, in both Chisel3 and Verilog, together with a software driver called test_driver.cc.
Also, we provide cmake scripts for building everything automatically and a config.json file for managing accelerator and simulation options.

Finally, the following snippet shows how a VTA design simulation, based on the add-by-one example, is invoked on TVM:
ctx = tvm.cpu(0)
a = tvm.nd.array(...) # input
b = tvm.nd.array(...) # output
tsim = tvm.module.load("libtsim.so", "vta-tsim")
f = tvm.get_global_func("tvm.vta.driver")
f(tsim, a, b)

This will facilitate continuous testing of novel VTA features, and invite more contribution to modify the VTA spec in the future. Cycle-accurate testing from high-level test scripts in Python is definitely the way forward for all of TVM's backend.
@kazum @ktabata it would be great to get your take on this since you took part in providing the SDAccel and AOCL support respectively in TVM. Having your take on this verilator-based simulation flow would be great.

@jroesch
There are two reasons why we have the two DPI modules in Verilog (Host and Memory):

To support either handwritten Verilog accelerators or generated Verilog from other languages different than Chisel3

Chisel3 does not support DPI, which is the "CFFI" of Verilog. However, Chisel3 does support Verilog inlining which is what we use for this so we don't duplicate code, see here

https://github.com/dmlc/tvm/pull/3010

authors: tmoreau89 tqchen vegaluisjose

repo: dmlc/tvm


text:
Add support for multiple OpenCL platforms

This PR allows having both SDAccel and other OpenCL platforms.
When the target device is 'sdaccel', the runtime tries to use the Xilinx toolchain and find FPGA devices.  If it fails to find SDAccel environment, it falls back to other OpenCL platforms so that unittests can test sdaccel backend.
When the target device is 'opencl', the runtime tries to find GPU and CPU devices.  The behavior is same as before.

@tqchen @tmoreau89 @comaniac Can you review this PR?  Thanks.

One question: will this flow support all three of (1) hw emulation, (2), sw emulation, (3), actual hardware execution on the FPGA.
In this case will the runtime always check for FPGA device, or only for case 3?
I can imagine launching an regular CPU instance with an FPGA development AMI, instead of an F1 instance, just to test OpenCL synthesis in software emulation/hardware emulation.
Another question, is how the different hardware FPGA backends should be handled by the runtime. Right now are we aiming to support only VUP9+ based F1 instances? Or are you also planning on supporting other development boards?
Overall, the PR looks acceptable, although I'm not an expert on the TVM runtime. Maybe @tqchen can give a final word on the changes.

One question: will this flow support all three of (1) hw emulation, (2), sw emulation, (3), actual hardware execution on the FPGA.
In this case will the runtime always check for FPGA device, or only for case 3?
I can imagine launching an regular CPU instance with an FPGA development AMI, instead of an F1 instance, just to test OpenCL synthesis in software emulation/hardware emulation.

This flow supports all of the three cases.  In the cases of (1) and (2), the Xilinx OpenCL platform tries to read emconfig.json and emulates devices written in the file.  We can use, e.g., non-F1 instances for development like that.

Another question, is how the different hardware FPGA backends should be handled by the runtime. Right now are we aiming to support only VUP9+ based F1 instances? Or are you also planning on supporting other development boards?

The current implementation should work for all Xilinx FPGAs which supports SDAccel.  I've not tested other FPGA environments than AWS F1 yet, though.

@kazum thank you for your answers. One more question, in the case of having a different SDAccel FPGA targets (i.e. different part numbers), what variable would we need to change to set the device number?

@kazum great, I think it's good that we keep in mind that by using the same SDAccel flow, we may be targeting multiple FPGA platforms.
I don't have anything to add, @tqchen if the changes to the run time look good, can you approve this PR?

Sorry for delayed review on this, see my comment.
The current way of supporting SDAccel is not desirable, because when both SDAccel and GPU are present, the global singleton can only be initialized to one platform.
If we indeed want to support sdaccel as a separate device as opencl -- which makes sense because the opencl for hardware is quite different. Then we might want to create a separate singleton for the SDAccel while reusing the current OpenCL runtime code.
This being said, we may not need the fake compile feature at the moment for the same reason that sdaccel is very different from OpenCL.
Seems that we just need a way to tell if sdaccel is available.

@tqchen Thanks for your comments.  I'm okay with separating SDAccel implementation from the OpenCL code because the current singleton architecture cannot support multiple FPGA devices in either way.
Then, I think of adding a base class for OpenCL devices, and re-implementing GPU and SDAccel supports upon it.  I think we don't need so big code changes for that.  Is it fine with you?
It's also okay to me that the fake compile feature is not necessary now.  I'll try to find a better way to test SDAccel codes for CI without SDAccel environment.

@kazum It sounds good.
About the multiple FPGA support, I think it is possible to support multiple FPGAs in current runtime, as long as these devices are all added. and you can use sdaccel(0) and sdaccel(1) to indicate devices

@tqchen Ah, I see.  I'll give it a try, thanks!

@tqchen I've pushed commits to separate SDAccel runtime and remove fake compile feature.  Can you check them out? About multiple FPGA support, I think of sending another patches later after we finish this PR.
The CI system fails to create test binaries now, but it looks like it refers to the old version of libtvm.so.  The same test can be passed successfully on my environment.

The CI system fails to create test binaries now, but it looks like it refers to the old version of libtvm.so.

So sorry, it was caused my bug, and I've fixed the problem.

@tqchen Thanks for your comments!  I've addressed them.

@tqchen Addressed, thanks!

thanks @kazum this is merged!

source: gharchive/pull-request

authors: kazum tmoreau89 tqchen

license_type: permissive

license_source: github-api


text:
[jvm-packages]error in compiler

Environment info
Operating System:win64
Compiler:tdm64
Package used (python/R/jvm/C++): jvm
xgboost version used: 0.7
There is an error when i compiler jvm-package , below is error information in maven
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for                                                                                ml.dmlc:xgboost4j:jar:0.7
[WARNING] 'build.plugins.plugin.version' for org.codehaus.mojo:exec-maven-plugin                                                                                is missing. @ line 54, column 19
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten t                                                                                he stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support buildin                                                                                g such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] xgboost-jvm
[INFO] xgboost4j
[INFO] xgboost4j-spark
[INFO] xgboost4j-flink
[INFO] xgboost4j-example
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building xgboost-jvm 0.8-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- scalastyle-maven-plugin:0.8.0:check (checkstyle) @ xgboost-jvm ---
[WARNING] sourceDirectory is not specified or does not exist value=C:\Users\jerry\Desktop\github\xgboost\jvm-packages\src\main\scala
[WARNING] testSourceDirectory is not specified or does not exist value=C:\Users\jerry\Desktop\github\xgboost\jvm-packages\src\test\scala
Processed 0 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 83 ms
[INFO]
[INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle) @ xgboost-jvm ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default) @ xgboost-jvm ---
[INFO] argLine set to "-javaagent:D:\Program Files\apache-maven-3.3.9\repo\org\jacoco\org.jacoco.agent\0.7.9\org.jacoco.agent-0.7.9-runtime.jar=destfile=C:\Users\jerry\Desktop\github\xgboost\jvm-packages\target\jacoco.exec"
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (default) @ xgboost-jvm ---
[WARNING]  Expected all dependencies to require Scala version: 2.11.8
[WARNING]  ml.dmlc:xgboost-jvm:0.8-SNAPSHOT requires scala version: 2.11.8
[WARNING]  org.scala-lang:scala-compiler:2.11.8 requires scala version: 2.11.8
[WARNING]  org.scala-lang.modules:scala-xml_2.11:1.0.4 requires scala version: 2.11.4
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ xgboost-jvm ---
[WARNING]  Expected all dependencies to require Scala version: 2.11.8
[WARNING]  ml.dmlc:xgboost-jvm:0.8-SNAPSHOT requires scala version: 2.11.8
[WARNING]  org.scala-lang:scala-compiler:2.11.8 requires scala version: 2.11.8
[WARNING]  org.scala-lang.modules:scala-xml_2.11:1.0.4 requires scala version: 2.11.4
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:add-source (scala-compile-first) @ xgboost-jvm ---
[INFO] Add Source directory: C:\Users\jerry\Desktop\github\xgboost\jvm-packages\src\main\scala
[INFO] Add Test Source directory: C:\Users\jerry\Desktop\github\xgboost\jvm-packages\src\test\scala
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (compile) @ xgboost-jvm ---
[WARNING]  Expected all dependencies to require Scala version: 2.11.8
[WARNING]  ml.dmlc:xgboost-jvm:0.8-SNAPSHOT requires scala version: 2.11.8
[WARNING]  org.scala-lang:scala-compiler:2.11.8 requires scala version: 2.11.8
[WARNING]  org.scala-lang.modules:scala-xml_2.11:1.0.4 requires scala version: 2.11.4
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (test-compile) @ xgboost-jvm ---
[WARNING]  Expected all dependencies to require Scala version: 2.11.8
[WARNING]  ml.dmlc:xgboost-jvm:0.8-SNAPSHOT requires scala version: 2.11.8
[WARNING]  org.scala-lang:scala-compiler:2.11.8 requires scala version: 2.11.8
[WARNING]  org.scala-lang.modules:scala-xml_2.11:1.0.4 requires scala version: 2.11.4
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ xgboost-jvm ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- jacoco-maven-plugin:0.7.9:report (report) @ xgboost-jvm ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO]
[INFO] --- maven-assembly-plugin:2.6:single (make-assembly) @ xgboost-jvm ---
[INFO] Assemblies have been skipped per configuration of the skipAssembly parameter.
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building xgboost4j 0.7
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- scalastyle-maven-plugin:0.8.0:check (checkstyle) @ xgboost4j ---
Processed 14 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 1243 ms
[INFO]
[INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle) @ xgboost4j ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default) @ xgboost4j ---
[INFO] argLine set to "-javaagent:D:\Program Files\apache-maven-3.3.9\repo\org\jacoco\org.jacoco.agent\0.7.9\org.jacoco.agent-0.7.9-runtime.jar=destfile=C:\Users\jerry\Desktop\github\xgboost\jvm-packages\xgboost4j\target\jacoco.exec"
[INFO]
[INFO] --- exec-maven-plugin:1.6.0:exec (native) @ xgboost4j ---
CMake Error at CMakeLists.txt:2 (project):
Failed to run MSBuild command:
MSBuild.exe

to get the value of VCTargetsPath:
-- Configuring incomplete, errors occurred!
See also "C:/Users/jerry/Desktop/github/xgboost/build/CMakeFiles/CMakeOutput.log".
building Java wrapper
cd ..
mkdir -p build
cd build
cmake .. -DUSE_S3:BOOL=OFF -DUSE_OPENMP:BOOL=ON -DPLUGIN_UPDATER_GPU:BOOL=OFF -DUSE_AZURE:BOOL=OFF -DUSE_HDFS:BOOL=OFF -DJVM_BINDINGS:BOOL=ON -G"Visual Studio 14 Win64"
Traceback (most recent call last):
File "create_jni.py", line 88, in
run("cmake .. " + " ".join(args) + maybe_generator)
File "create_jni.py", line 51, in run
subprocess.check_call(command, shell=True, **kwargs)
File "D:\ProgramData\Anaconda2\lib\subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'cmake .. -DUSE_S3:BOOL=OFF -DUSE_OPENMP:BOOL=ON -DPLUGIN_UPDATER_GPU:BOOL=OFF -DUSE_AZURE:BOOL=OFF -DUSE_HDFS:BOOL=OFF -DJVM_BINDINGS:BOOL=ON -G"Visual Studio 14 Win64"' returned non-zero exit status 1
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:804)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:751)
at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:313)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] xgboost-jvm ........................................ SUCCESS [  3.562 s]
[INFO] xgboost4j .......................................... FAILURE [  2.024 s]
[INFO] xgboost4j-spark .................................... SKIPPED
[INFO] xgboost4j-flink .................................... SKIPPED
[INFO] xgboost4j-example .................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.735 s
[INFO] Finished at: 2018-01-06T02:08:19+08:00
[INFO] Final Memory: 23M/289M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (native) on project xgboost4j: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :xgboost4j

same issue

I solved this problem. Firstly make sure you installed MSBuild on your computer.  You can google how to install.   Don't forget to set the directory of MSBuild to your system path variable (user variable for .. add it to the "path"). Now if you test to install xgboost again,  MSBuild can be find, but the version may be not correct. I cannot remember well, but you can google how to change the msbuild version. Hope this can help.

source: gharchive/issue

repo: dmlc/xgboost


text:
[Roadmap] 1.4.0 Roadmap

@dmlc/xgboost-committer Please add your items here by editing this post. Let's ensure that

Each item has to be associated with a ticket
Major design/refactoring are associated with a RFC before committing the code
Blocking issue must be marked as blocking
Breaking change must be marked as breaking

For other contributors who have no permission to edit the post, please comment here about what you think should be in 1.4.0.
Main

[x] Initial support for Python type hints. (#6496, #6519)
[ ] Improve ranking implementation. (#6450)
[x] Add base margin to evaluation dataset. (#6591)
[x] Further development for inplace prediction. (#6304 #6653)
[ ] Further development on categorical data support. (#6503)
[ ] Add score tree interval to early stopping. (#5238, #5090)
[x] Remove saving binary format for memory snapshot. (#6513)
[ ] Staged predict (https://github.com/dmlc/xgboost/issues/5006)
[x] Global configuration.  (#6414)
[x] Training continuation with early stopping. (#6506)
[ ] Add use_rmm flag to global configuration (#6656)
[x] Deprecate ntree_limit in Python.
[ ] Use inplace_predict for sklearn. (#6718)
[x] Make all prediction functions for gbtree thread safe. (#6648)

Dask

[x] Make dask scikit-learn interface feature complete. (#6471, #6602)

[x] Add ranking for dask. (#6576)
[x] Add other random forest interfaces. (#6602)
[x] Use inplace prediction for dask. (#6668)

[ ] Support training multiple models in parallel using dask.
[x] Optimize prediction performance. (#6614)
[x] Support shap value computation. (#6645)

For brief notes, at 1.4, dask interface should be feature complete, categorical data support for GPU is ready for public testing and inplace prediction will be more mature.

https://github.com/dmlc/xgboost/pull/6507

I want to propose to replace 'approx' to 'hist when tree_method is set to 'auto' on CPU. I observed https://github.com/dmlc/xgboost/pull/5178 was about this, but it's closed.
hist is faster than approx and my observations that even accuracy is better/on par. GPUs also have only 'hist' method, not 'approx'.
Do we have any concerns not to do this?
CC: @trivialfis, @hcho3, @ShvetsKS

I will put up some documents on theoretical aspect of various tree methods, then we can decide together.

in https://github.com/dmlc/xgboost/pull/6564

in https://github.com/dmlc/xgboost/pull/6564

@trivialfis, do we need to run experiments to decide, probably?

@trivialfis, do we need to run experiments to decide, probably?

@SmirnovEgorRu I don't have objection of changing the default in this or next release.  I mentioned there's a huge refactor for CPU implementations to @ShvetsKS .  I would like to see some parts of it merged before making the change so we can make some fair comparison.  Will come back to it after sorting out issues in dask interface. (which should be quite fast as most of the features are now supported).

@SmirnovEgorRu I don't have objection of changing the default in this or next release.  I mentioned there's a huge refactor for CPU implementations to @ShvetsKS .  I would like to see some parts of it merged before making the change so we can make some fair comparison.  Will come back to it after sorting out issues in dask interface. (which should be quite fast as most of the features are now supported).

I closed the PR you referenced because I couldn't get all tests passing, I think even if we decided to change the default now, we still have some blockers to track down.  So refactoring first might help making the change clearer and easier.

I closed the PR you referenced because I couldn't get all tests passing, I think even if we decided to change the default now, we still have some blockers to track down.  So refactoring first might help making the change clearer and easier.

Hi @ShvetsKS @SmirnovEgorRu I have been trying to refactor the CPU code for categorical data support based on the efficient CPU Hist code.  I found the on URL dataset the cpu hist is slower than approx.  It's not a conventional dataset as it's unusually wide and sparse.  Just curious if you have plan on optimizing it.

Hi @ShvetsKS @SmirnovEgorRu I have been trying to refactor the CPU code for categorical data support based on the efficient CPU Hist code.  I found the on URL dataset the cpu hist is slower than approx.  It's not a conventional dataset as it's unusually wide and sparse.  Just curious if you have plan on optimizing it.

The approx implementation is parallelizing on features with dynamic scheduling, so it has an advantage on these kind of datasets.

The approx implementation is parallelizing on features with dynamic scheduling, so it has an advantage on these kind of datasets.

@trivialfis, yep, we are thinking how to tune wide data sets as well. I suppose we can outperform approx with hist on URL.

@trivialfis, yep, we are thinking how to tune wide data sets as well. I suppose we can outperform approx with hist on URL.

"Support training multiple models in parallel using dask". Does this include cross-validation with early stopping?

@Denisevi4 No, it's for running multiple training sessions on a single cluster simultaneously.  But it's a basic requirement for cv.

#6731

@hcho3 I would like to get the 1.4 out once we got AUC re-implemented.  I can try fixing the gamma metric if the AUC re-implementation goes well.

I will branch out next week.

I will branch out next week.

Will you fix other metrics (gamma-nloglik, logloss)?

Yeah, I will take a deeper look into them this weekend.

@Roffild Will reply on the original thread: https://github.com/dmlc/xgboost/issues/6731

1.4 is out, submit status will be on https://github.com/dmlc/xgboost/issues/6793 .

authors: Denisevi4 Roffild SmirnovEgorRu trivialfis


text:
[jvm-packages] Release dmatrix when no longer needed

When using xgboost4j-spark I had executors getting killed much more
often than i would expect by yarn for overrunning their memory limits,
based on the memoryOverhead provided. It looks like a significant
amount of this is because dmatrix's were being created but not release
as they were only released when the GC decided it was time to
cleanup the references.
Rather than waiting for the GC, release the DMatrix's when we know
they are no longer necessary.

LGTM, waiting for travis, thanks!

merged, thanks

authors: CodingCat ebernhardson


text:
feat: Logo design added

This pull request adds new logo designs to the project. The logos are in the public folder, and there are a few versions to choose from:

Logo-01.jpg
Logo-1.png
Logo-2.png
Logo-3.png
These logos can be used for the app or any branding needs.

Let me know if any changes are needed!

I love it! 🫶
Thank you for contributing @shawaiz-khan!

Thank You! please make this PR hacktoberfest-accepted.

No, thank you Mr. @shawaiz-khan!
As part of improving and making this project a enjoyable one for all, I ask that you share your opinions in our discussions section.
This will not only help the project grow but it will help all of us grow as a people.
Thanks again, @shawaiz-khan and keep doing you!

No, thank you Mr. @shawaiz-khan!
As part of improving and making this project an enjoyable one for all, I ask that you share your opinions in our discussions section.
This will not only help the project grow but it will help all of us grow as a people.
Thanks again, @shawaiz-khan and keep doing you!

Sure

authors: dmotts shawaiz-khan

repo: dmotts/yt-to-cloud-uploader

license: MIT


text:
🛑 Google is down

In ae73d34, Google (https://www.google.com) was down:

HTTP code: 429
Response time: 421 ms

Resolved: Google is back up in b2cb65b.

repo: dmuinoo/uptime


text:
Migrate LogCollect CASTOR endpoint to CTA

Impact of the new feature
WMAgent
Is your feature request related to a problem? Please describe.
CASTOR has been recently put in read-only mode and it's getting decommissioned in favor of the CERN CTA tape technology.
That means WMAgent components that depend on CERN CASTOR have also to be migrated/adapted to this new technology.
Describe the solution you'd like
Discussion with experts is still in the initial phase, however some of the information provided by Katy (and her contacts at CERN are):
The production instance name will be eosctacms.cern.ch
The Rucio prefix should be /eos/ctacms/archive but I will confirm when the final import is finished.
So the PFN would be : root://eosctacms.cern.ch//eos/ctacms/archive

It's still not clear whether xrdcp will work and what authentication/authorization mechanism will be required. So this needs further discussion.
Describe alternatives you've considered
Another alternative suggested by Eric V. would be to manage those logs with Rucio, or perhaps use the rucio upload mechanism to write these files to the CTA storage. Another topic that requires further discussion.
Additional context
Just to add some context, the current behaviour of such LogCollect jobs is the following:

if possible, use edmCopyUtil from the CMSSW release (thus CMSSW needs to be bootstrapped) to stage files in
copy files from /store/unmerged/logs into the worker node scratch area
add all the successful logs in the scratch area to a big tarball (delete every one of them as soon as they get added to the tarball)
write the tarball to CASTOR (/store/logs/prod), abort the job if any failure happens
try to write the tarball CERN EOS in a best effort mode (failures are discarded)
last but not least, delete every single file that we managed to copy from the /store/unmerged/logs to the scratch area. These are deleted from /store/unmerged/logs

FIXME TODO:
While looking at the code, there seems to be a bug on the file deletion from the scratch area (not every log is deleted, but only the last one in the loop):
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/Steps/Executors/LogCollect.py#L204
and a second problem - or improvement - would be to delete all the logArchive files passed to this job, not only those that we managed to copy to the scratch area:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/Steps/Executors/LogCollect.py#L262

Just an update on this issue. I have performed some interactive tests with rucio upload, also discussed a few things with Katy and Eric, here is a summary and the current status.
Eric has created a new scope, called logs, where we can write these logs to.
A new Rucio account has also been created for these activities, the account name is wma_logs.
Last but not least, given that grid jobs run with my personnel proxy, Eric mapped my personnel DN to the wma_logs rucio account.
Here is the outcome of the interactive tests I was running on lxplus.
Setting up the rucio client environment:
source /cvmfs/cms.cern.ch/cmsset_default.sh
voms-proxy-init -voms cms -rfc -valid 192:00
source /cvmfs/cms.cern.ch/rucio/setup.sh
export RUCIO_ACCOUNT=wma_logs

then accessing the directory with the log tarball, and writing it to CERN EOS:
amaltaro@lxplus783:~/workspace $ rucio -v upload --scope logs --rse T2_CH_CERN --lifetime 86400  --name /store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar LogCollect-cmswn2292-1-logs.tar
...
2021-01-13 15:08:12,391	INFO	Preparing upload for file LogCollect-cmswn2292-1-logs.tar
...
2021-01-13 15:08:12,572	INFO	Successfully added replica in Rucio catalogue at T2_CH_CERN
2021-01-13 15:08:12,689	INFO	Successfully added replication rule at T2_CH_CERN

...
2021-01-13 15:08:14,253	INFO	Trying upload with root to T2_CH_CERN
2021-01-13 15:08:14,253	DEBUG	uploadclient.py	Processing upload with the domain: wan
2021-01-13 15:08:14,439	DEBUG	uploadclient.py	The PFN created from the LFN: root://eoscms.cern.ch:1094//eos/cms/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar
2021-01-13 15:08:14,763	DEBUG	utils.py	put: Attempt 1
2021-01-13 15:08:15,023	INFO	Successful upload of temporary file. root://eoscms.cern.ch:1094//eos/cms/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar.rucio.upload
2021-01-13 15:08:15,023	DEBUG	uploadclient.py	skip_upload_stat=False
2021-01-13 15:08:15,024	DEBUG	uploadclient.py	stat: pfn=root://eoscms.cern.ch:1094//eos/cms/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar.rucio.upload
2021-01-13 15:08:15,026	DEBUG	uploadclient.py	Filesize: Expected=225280 Found=225280
2021-01-13 15:08:15,027	DEBUG	uploadclient.py	Checksum: Expected=675eedbd Found=675eedbd
2021-01-13 15:08:15,027	DEBUG	uploadclient.py	Renaming file root://eoscms.cern.ch:1094//eos/cms/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar.rucio.upload to root://eoscms.cern.ch:1094//eos/cms/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar
2021-01-13 15:08:15,030	DEBUG	uploadclient.py	Upload done.
2021-01-13 15:08:15,031	INFO	Successfully uploaded file LogCollect-cmswn2292-1-logs.tar

few observations are:

the file gets properly renamed to whatever we pass in the end of the --name parameter
rucio upload first adds that file replica to the Rucio catalogue, then it makes the transfer
--lifetime 86400 is provided in this test. Help pages say it's the lifetime of the rule(?) Did we also get a rule created here? Is the file eligible for deletion after that lifetime?

What's left to be investigated is how we can create a container (or block) and attach these replicas to them. The agent does not keep track of the output of LogCollect jobs, so making such operation on the agent side would be quite intrusive with many changes.
@ericvaandering @KatyEllis FYI

Just an update on this issue. I have performed some interactive tests with rucio upload, also discussed a few things with Katy and Eric, here is a summary and the current status.
Eric has created a new scope, called logs, where we can write these logs to.
A new Rucio account has also been created for these activities, the account name is wma_logs.
Last but not least, given that grid jobs run with my personnel proxy, Eric mapped my personnel DN to the wma_logs rucio account.
Here is the outcome of the interactive tests I was running on lxplus.
Setting up the rucio client environment:
source /cvmfs/cms.cern.ch/cmsset_default.sh
voms-proxy-init -voms cms -rfc -valid 192:00
source /cvmfs/cms.cern.ch/rucio/setup.sh
export RUCIO_ACCOUNT=wma_logs

then accessing the directory with the log tarball, and writing it to CERN EOS:
amaltaro@lxplus783:~/workspace $ rucio -v upload --scope logs --rse T2_CH_CERN --lifetime 86400  --name /store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar LogCollect-cmswn2292-1-logs.tar
...
2021-01-13 15:08:12,391	INFO	Preparing upload for file LogCollect-cmswn2292-1-logs.tar
...
2021-01-13 15:08:12,572	INFO	Successfully added replica in Rucio catalogue at T2_CH_CERN
2021-01-13 15:08:12,689	INFO	Successfully added replication rule at T2_CH_CERN

...
2021-01-13 15:08:14,253	INFO	Trying upload with root to T2_CH_CERN
2021-01-13 15:08:14,253	DEBUG	uploadclient.py	Processing upload with the domain: wan
2021-01-13 15:08:14,439	DEBUG	uploadclient.py	The PFN created from the LFN: root://eoscms.cern.ch:1094//eos/cms/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar
2021-01-13 15:08:14,763	DEBUG	utils.py	put: Attempt 1
2021-01-13 15:08:15,023	INFO	Successful upload of temporary file. root://eoscms.cern.ch:1094//eos/cms/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar.rucio.upload
2021-01-13 15:08:15,023	DEBUG	uploadclient.py	skip_upload_stat=False
2021-01-13 15:08:15,024	DEBUG	uploadclient.py	stat: pfn=root://eoscms.cern.ch:1094//eos/cms/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar.rucio.upload
2021-01-13 15:08:15,026	DEBUG	uploadclient.py	Filesize: Expected=225280 Found=225280
2021-01-13 15:08:15,027	DEBUG	uploadclient.py	Checksum: Expected=675eedbd Found=675eedbd
2021-01-13 15:08:15,027	DEBUG	uploadclient.py	Renaming file root://eoscms.cern.ch:1094//eos/cms/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar.rucio.upload to root://eoscms.cern.ch:1094//eos/cms/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar
2021-01-13 15:08:15,030	DEBUG	uploadclient.py	Upload done.
2021-01-13 15:08:15,031	INFO	Successfully uploaded file LogCollect-cmswn2292-1-logs.tar

few observations are:

the file gets properly renamed to whatever we pass in the end of the --name parameter
rucio upload first adds that file replica to the Rucio catalogue, then it makes the transfer
--lifetime 86400 is provided in this test. Help pages say it's the lifetime of the rule(?) Did we also get a rule created here? Is the file eligible for deletion after that lifetime?

What's left to be investigated is how we can create a container (or block) and attach these replicas to them. The agent does not keep track of the output of LogCollect jobs, so making such operation on the agent side would be quite intrusive with many changes.
@ericvaandering @KatyEllis FYI

Yeah, there is a rule: -bash-4.2$ rucio list-rules logs:/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar
ID                                ACCOUNT    SCOPE:NAME                                                                                STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)        CREATED (UTC)

4a7f0397a3f34ca0a6b13214c6bb4bdc  wma_logs   logs:/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar  OK[1/0/0]               T2_CH_CERN               1  2021-01-14 14:08:12  2021-01-13 14:08:12
Shall we coordinate by Slack with setting up a subscription which copies these to CTA?

Yeah, there is a rule: -bash-4.2$ rucio list-rules logs:/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar
ID                                ACCOUNT    SCOPE:NAME                                                                                STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)        CREATED (UTC)

4a7f0397a3f34ca0a6b13214c6bb4bdc  wma_logs   logs:/store/logs/prod/2021/01/WMAgent/amaltaro_TaskChain_ProdMinBias_Jan2021_Val_210112_212625_3216/LogCollect-cmswn2292-2-logs.tar  OK[1/0/0]               T2_CH_CERN               1  2021-01-14 14:08:12  2021-01-13 14:08:12
Shall we coordinate by Slack with setting up a subscription which copies these to CTA?

Another update to this thread, after what we discussed in yesterday's Rucio transition meeting.
In addition to running the rucio upload command to transfer (and register) the log tarball to the CERN EOS storage, we will also try to create a container DID and attach the tarball file to it. These 3 operations will all be done from the worker node.
Container DID will likely be named after the workflow name, thus we should expect and properly handle the DataIdentifierAlreadyExists exception.

Another update to this thread, after what we discussed in yesterday's Rucio transition meeting.
In addition to running the rucio upload command to transfer (and register) the log tarball to the CERN EOS storage, we will also try to create a container DID and attach the tarball file to it. These 3 operations will all be done from the worker node.
Container DID will likely be named after the workflow name, thus we should expect and properly handle the DataIdentifierAlreadyExists exception.

Defined as low prio, from the point of view of the P&R team (Hasan)

And from the question raised during the O&C weekly meeting, here is a breakdown of how much space these logs are meant to take in CERN CTA:
amaltaro@vocms0275:~ $ du -sh /eos/cms/store/logs/prod/2022/01/WMAgent/
7.3T	/eos/cms/store/logs/prod/2022/01/WMAgent/
amaltaro@vocms0275:~ $ du -sh /eos/cms/store/logs/prod/2022/02/WMAgent/
6.1T	/eos/cms/store/logs/prod/2022/02/WMAgent/
amaltaro@vocms0275:~ $ du -sh /eos/cms/store/logs/prod/2022/03/WMAgent/
4.9T	/eos/cms/store/logs/prod/2022/03/WMAgent/

considering the past 3 months. This EOS area has exactly the same tarballs that we would be trying to write to CTA.

Having not heard any requests to get this feature ready for data taking, and from a feedback from Danilo that the current data volume is okay to be kept in EOS, I am downgrading it (removing the Data taking label) and we can work on it in the quarters to come.

authors: amaltaro ericvaandering

repo: dmwm/WMCore