text:
Flash Attention benchmarks fail on AMD Instinct MI210 (gfx90a)

I am currently reworking the flash attention benchmarking script provided here. (This script was used to produce the data for the plots of the A100 and H100 benchmarks provided in the README.)
The adjusted script runs fine on gfx90a for the default flash attention 2 and pytorch benchmarks, but fails when I run the Triton benchmark (Memory access fault). I have isolated the Triton benchmark into its own script [1]. For clarity, the benchmarking script loops through various parameters (see the for loops in the relevant script), such as False/True for causal masks, 64/128 for head dimension size, and the tuples (32, 512), (16, 1024), (8, 2048), (4, 4096), (2, 8192), (1, 16384) which provide the batch size and sequence length, respectively. The metric being benchmarked is TFLOPs/s.
The benchmark fails at causal=False, headdim=64, batch_size, seqlen = (2, 8192). This is not a quantity of memory issue, the torch profiler indicates that only ~10/64GB is being used at this point. Logging indicates that the memory access fault occurs in this block:
try:
    _, b0 = time_fwd_bwd(
        attention_triton, q, k, v, causal, headdim**(-0.5),
        True, repeats=repeats, verbose=False
)

time_fwd_bwd eventually traces back to the backward method of triton.ops.flash_attention (inherited from torch.autograd.Function). That backward method is provided here:
@staticmethod
    def backward(ctx, do):
        capability = torch.cuda.get_device_capability()
        MMA_V3 = capability[0] >= 9
        BLOCK = 128

        if is_hip():
            # Bwd pass runs out of shared memory on HIP with larger block size.
            BLOCK = 64

        q, k, v, o, L = ctx.saved_tensors
        sequence_parallel = ctx.sequence_parallel
        seq_len_kv = k.shape[2]
        do = do.contiguous()
        if sequence_parallel:
            replicas = cdiv(seq_len_kv, BLOCK)
            new_dq_shape = (replicas, ) + q.shape
            dq = torch.zeros(new_dq_shape, device=q.device, dtype=q.dtype)
        else:
            dq = torch.zeros_like(q, dtype=q.dtype)
        dk = torch.empty_like(k)
        dv = torch.empty_like(v)
        delta = torch.empty_like(L)
        _bwd_preprocess[(cdiv(q.shape[2], BLOCK) * ctx.grid[1], )](
            o,
            do,
            delta,
            BLOCK_M=BLOCK,
            D_HEAD=ctx.BLOCK_DMODEL,
        )
        _bwd_kernel[(ctx.grid[1], cdiv(seq_len_kv, BLOCK) if sequence_parallel else 1)](
            q, k, v, ctx.sm_scale,  #
            o, do,  #
            dq, dk, dv,  #
            L,  #
            delta,  #
            o.numel(), q.stride(0), q.stride(1), q.stride(2), q.stride(3),  #
            k.stride(0), k.stride(1), k.stride(2), k.stride(3),  #
            v.stride(0), v.stride(1), v.stride(2), v.stride(3),  #
            q.shape[0], q.shape[1], q.shape[2],  #
            q.shape[0] * q.shape[1] * q.shape[2],  #
            cdiv(seq_len_kv, BLOCK) * q.shape[0] * q.shape[1] * q.shape[2],  #
            BLOCK_M=BLOCK, BLOCK_N=BLOCK,  #
            BLOCK_DMODEL=ctx.BLOCK_DMODEL,  #
            SEQUENCE_PARALLEL=sequence_parallel,  #
            CAUSAL=ctx.causal,  #
            MMA_V3=MMA_V3,  #
            num_warps=8,  #
            num_stages=1  #
        )

        if len(dq.shape) == 5:
            dq = dq.sum(dim=0)
        return dq, dk, dv, None, None, None

The only difference between this and the prior try/except block that works correctly is that sequence_parallel is set to True. The memory access fault consistently occurs the third time that backward is called with the parameters described above.
I am not quite sure where to investigate from here. For the sake of experimentation, I have varied if is_hip(): BLOCK=64 to other values, but this just coerces it into failing earlier. (e.g., BLOCK=32 or BLOCK=128 fails at seqlen=4096).
Is there any information or intuition into what might be causing this?
For reference, the full error message is: Memory access fault by GPU node-2 (Agent handle: 0x6175bb0) on address 0x7b5a69201000. Reason: Unknown. Aborted (core dumped). Of course, the actual memory locations vary per run.
[1] Triton benchmark script
import math
import torch
import torch.nn.functional as F
import numpy as np

from einops import rearrange
from flash_attn.utils.benchmark import benchmark_fwd_bwd

from triton.ops.flash_attention import attention as attention_triton

def flops(batch, seqlen, headdim, nheads, causal, mode="fwd"):
    assert mode in ["fwd", "bwd", "fwd_bwd"]
    f = 4 * batch * seqlen**2 * nheads * headdim // (2 if causal else 1)
    return f if mode == "fwd" else (2.5 * f if mode == "bwd" else 3.5 * f)

def efficiency(flop, time):
    return (flop / time / 10**12) if not math.isnan(time) else 0.0

def attention_pytorch(qkv, dropout_p=0.0, causal=True):
    """
    Arguments:
        qkv: (batch_size, seqlen, 3, nheads, head_dim)
        dropout_p: float
    Output:
        output: (batch_size, seqlen, nheads, head_dim)
    """
    batch_size, seqlen, _, nheads, d = qkv.shape
    q, k, v = qkv.unbind(dim=2)
    q = rearrange(q, 'b t h d -> (b h) t d')
    k = rearrange(k, 'b s h d -> (b h) d s')
    softmax_scale = 1.0 / math.sqrt(d)
    # Preallocate attn_weights for `baddbmm`
    scores = torch.empty(batch_size * nheads, seqlen, seqlen, dtype=qkv.dtype, device=qkv.device)
    scores = rearrange(torch.baddbmm(scores, q, k, beta=0, alpha=softmax_scale),
                       '(b h) t s -> b h t s', h=nheads)
    if causal:
        # "triu_tril_cuda_template" not implemented for 'BFloat16'
        # So we have to construct the mask in float
        causal_mask = torch.triu(torch.full((seqlen, seqlen), -10000.0, device=scores.device), 1)
        # TD [2022-09-30]: Adding is faster than masked_fill_ (idk why, just better kernel I guess)
        scores = scores + causal_mask.to(dtype=scores.dtype)
    attention = torch.softmax(scores, dim=-1)
    attention_drop = F.dropout(attention, dropout_p)
    output = torch.einsum('bhts,bshd->bthd', attention_drop , v)
    return output.to(dtype=qkv.dtype)

def time_fwd_bwd(func, *args, **kwargs):
    # https://github.com/Dao-AILab/flash-attention/blob/5018ac6ac531aabdb05c8af1ba3d98a2235bdbde/flash_attn/utils/benchmark.py#L117
    time_f, time_b = benchmark_fwd_bwd(func, *args, **kwargs)
    result = time_f[1].mean, time_b[1].mean
    return result

repeats = 30
device = 'cuda'
dtype = torch.float16

bs_seqlen_vals = [(32, 512), (16, 1024), (8, 2048), (4, 4096), (2, 8192), (1, 16384)]
causal_vals = [False, True]
headdim_vals = [64, 128]
dim = 2048
dropout_p = 0.0

methods = ["Triton"]

time_f = {}
time_b = {}
time_f_b = {}
speed_f = {}
speed_b = {}
speed_f_b = {}

for causal in causal_vals:
    for headdim in headdim_vals:
        for batch_size, seqlen in bs_seqlen_vals:
            config = (causal, headdim, batch_size, seqlen)
            nheads = dim // headdim
            q, k, v = [torch.randn(batch_size, nheads, seqlen, headdim, device=device, dtype=dtype,
                                requires_grad=True) for _ in range(3)]
            # Try both values of sequence_parallel and pick the faster one
            try:
                f, b = time_fwd_bwd(
                    attention_triton, q, k, v, causal, headdim**(-0.5),
                    False, repeats=repeats, verbose=False
                )
            except:
                f, b = float('nan'), float('inf')
            try:
                _, b0 = time_fwd_bwd(
                    attention_triton, q, k, v, causal, headdim**(-0.5),
                    True, repeats=repeats, verbose=False
                )
            except:
                b0 = float('inf')
            time_f[config, "Triton"] = f
            time_b[config, "Triton"] = min(b, b0) if min(b, b0) < float('inf') else float('nan')

            print(f"### causal={causal}, headdim={headdim}, batch_size={batch_size}, seqlen={seqlen} ###")
            for method_index, method in enumerate(methods):
                time_f_b[config, method] = time_f[config, method] + time_b[config, method]
                speed_f[config, method] = efficiency(
                    flops(batch_size, seqlen, headdim, nheads, causal, mode="fwd"),
                    time_f[config, method]
                )
                speed_b[config, method] = efficiency(
                    flops(batch_size, seqlen, headdim, nheads, causal, mode="bwd"),
                    time_b[config, method]
                )
                speed_f_b[config, method] = efficiency(
                    flops(batch_size, seqlen, headdim, nheads, causal, mode="fwd_bwd"),
                    time_f_b[config, method]
                )
                print(
                    f"{method} fwd: {speed_f[config, method]:.2f} TFLOPs/s, "
                    f"bwd: {speed_b[config, method]:.2f} TFLOPs/s, "
                    f"fwd + bwd: {speed_f_b[config, method]:.2f} TFLOPs/s"
                )

Thanks for reporting the issue! Maybe we have an OOB issue somewhere. A few quick tips on helping debugging: you can set AMD_LOG_LEVEL to 3 to see if there are more detailed error information. You can also try rocgdb to run your python script.

I have traced the breaking point as follows:
From the original script, repeats is eventually passed to flash_attn.utils.benchmark.benchmark_fwd_bwd in the line m = t.timeit(repeats). The benchmark_fwd_bwd function calls benchmark_backward from the same module. [1]
The paper trails then leads to torch.utils.benchmark.utils.timer.py (link), which gets its definition of timeit from timeit_template.cpp. [2]
Previously I had stated that the failure occurs during the third call of backward. I realize now that it occurs during the second call to backwards. I have found this to be consistent regardless of the value of repeats. (For reasons I do not totally understand, even if repeats is set to a low value (say, 1 or 2) it will still call forward and backward four times.)
Although they are lengthy, I will attach the outputs of with AMD_LOG_LEVEL=3. The first success call to backward [3] and the one at which it fails. [4]
[1] benchmark_backward
def benchmark_backward(
    fn,
    *inputs,
    grad=None,
    repeats=10,
    desc="",
    verbose=True,
    amp=False,
    amp_dtype=torch.float16,
    **kwinputs,
):
    """Use Pytorch Benchmark on the backward pass of an arbitrary function."""
    if verbose:
        print(desc, "- Backward pass")
    with torch.autocast(device_type="cuda", dtype=amp_dtype, enabled=amp):
        y = fn(*inputs, **kwinputs)
        if type(y) is tuple:
            y = y[0]
    if grad is None:
        grad = torch.randn_like(y)
    else:
        if grad.shape != y.shape:
            raise RuntimeError("Grad shape does not match output shape")

    def f(*inputs, y, grad):
        # Set .grad to None to avoid extra operation of gradient accumulation
        for x in inputs:
            if isinstance(x, torch.Tensor):
                x.grad = None
        y.backward(grad, retain_graph=True)

    t = benchmark.Timer(
        stmt="f(*inputs, y=y, grad=grad)",
        globals={"f": f, "inputs": inputs, "y": y, "grad": grad},
        num_threads=torch.get_num_threads(),
    )
    m = t.timeit(repeats)
    if verbose:
        print(m)
    return t, m

[2] timeit_template.cpp
/* C++ template for Timer.timeit

This template will be consumed by `cpp_jit.py`, and will replace:
    `GLOBAL_SETUP_TEMPLATE_LOCATION`,
    `SETUP_TEMPLATE_LOCATION`
      and
    `STMT_TEMPLATE_LOCATION`
sections with user provided statements.
*/
#include <chrono>

#include <c10/util/irange.h>
#include <torch/csrc/utils/pybind.h>
#include <pybind11/pybind11.h>
#include <torch/extension.h>

// Global setup. (e.g. #includes)
// GLOBAL_SETUP_TEMPLATE_LOCATION

double timeit(int n) {
  pybind11::gil_scoped_release no_gil;

  // Setup
  // SETUP_TEMPLATE_LOCATION

  {
    // Warmup
    // STMT_TEMPLATE_LOCATION
  }

  // Main loop
  auto start_time = std::chrono::high_resolution_clock::now();
  for (const auto loop_idx : c10::irange(n)) {
    (void)loop_idx;
    // STMT_TEMPLATE_LOCATION
  }
  auto end_time = std::chrono::high_resolution_clock::now();
  return std::chrono::duration<double>(end_time - start_time).count();
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
  m.def("timeit", &timeit);
}

[3] successful call to backward
INFO:triton.ops.flash_attention:backward
:3:hip_device_runtime.cpp   :623 : 4815821497786 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3e7c )
:3:hip_device_runtime.cpp   :631 : 4815821497794 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821497822 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3e7c )
:3:hip_device_runtime.cpp   :631 : 4815821497828 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device.cpp           :477 : 4815821497837 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevicePropertiesR0000 ( 0x70536bbc37a0, 0 )
:3:hip_device.cpp           :578 : 4815821497844 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevicePropertiesR0000: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821497977 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc36d4 )
:3:hip_device_runtime.cpp   :631 : 4815821497986 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821497991 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3624 )
:3:hip_device_runtime.cpp   :631 : 4815821497997 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821498002 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc346c )
:3:hip_device_runtime.cpp   :631 : 4815821498008 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821498028 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821498035 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2f7c )
:3:hip_device_runtime.cpp   :631 : 4815821498043 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_graph.cpp            :826 : 4815821498057 us: [pid:2429112 tid:0x70536bbc5640]  hipStreamIsCapturing ( stream:<null>, 0x70536bbc3010 )
:3:hip_graph.cpp            :827 : 4815821498064 us: [pid:2429112 tid:0x70536bbc5640] hipStreamIsCapturing: Returned hipSuccess :
:3:hip_memory.cpp           :599 : 4815821498074 us: [pid:2429112 tid:0x70536bbc5640]  hipMalloc ( 0x70536bbc30b8, 8589934592 )
:3:rocdevice.cpp            :2349: 4815821498554 us: [pid:2429112 tid:0x70536bbc5640] device=0x6fee8c0, freeMem_ = 0xdddc00000
:3:hip_memory.cpp           :601 : 4815821498568 us: [pid:2429112 tid:0x70536bbc5640] hipMalloc: Returned hipSuccess : 0x705165a00000: duration: 494 us
:3:hip_device_runtime.cpp   :653 : 4815821498580 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821498585 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821498593 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821498598 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821498609 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3914 )
:3:hip_device_runtime.cpp   :631 : 4815821498614 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821498622 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc37b4 )
:3:hip_device_runtime.cpp   :631 : 4815821498631 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821498647 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2ecc )
:3:hip_device_runtime.cpp   :631 : 4815821498652 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821498662 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {1048576,1,1}, {256,1,1}, 0, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821498667 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821498676 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {4294967152,4294967295,0}, {4126967784,28757,1807495048}, 0x70536bbc2f80, 0x70536bbc2f70 )
:3:hip_platform.cpp         :237 : 4815821498681 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821498691 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a988e9d0, {1048576,1,1}, {256,1,1}, 0x70536bbc2fa0, 0, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821498707 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:1073741824
:3:rocvirtual.cpp           :798 : 4815821498712 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = val:
:3:rocvirtual.cpp           :723 : 4815821498718 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x705165a00000 obj:[0x705165a00000-0x705365a00000]
:3:rocvirtual.cpp           :3016: 4815821498729 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIN3c104HalfEEENS_6detail5ArrayIPcLi1EEEEEviT0_T1_
:3:hip_module.cpp           :670 : 4815821498736 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821498741 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821498748 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2ecc )
:3:hip_device_runtime.cpp   :631 : 4815821498754 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821498763 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {1048576,1,1}, {256,1,1}, 0, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821498768 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821498776 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {1048576,1,1}, {256,1,1}, 0x70536bbc2f80, 0x70536bbc2f70 )
:3:hip_platform.cpp         :237 : 4815821498783 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821498793 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a988e9d0, {1048576,1,1}, {256,1,1}, 0x70536bbc2fa0, 0, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821498801 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:1073741824
:3:rocvirtual.cpp           :798 : 4815821498806 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = val:
:3:rocvirtual.cpp           :723 : 4815821498812 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x7051e5a00000 obj:[0x705165a00000-0x705365a00000]
:3:rocvirtual.cpp           :3016: 4815821498817 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIN3c104HalfEEENS_6detail5ArrayIPcLi1EEEEEviT0_T1_
:3:hip_module.cpp           :670 : 4815821498828 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821498833 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821498841 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2ecc )
:3:hip_device_runtime.cpp   :631 : 4815821498847 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821498855 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {1048576,1,1}, {256,1,1}, 0, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821498863 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821498877 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {1048576,1,1}, {256,1,1}, 0x70536bbc2f80, 0x70536bbc2f70 )
:3:hip_platform.cpp         :237 : 4815821498883 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821498893 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a988e9d0, {1048576,1,1}, {256,1,1}, 0x70536bbc2fa0, 0, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821498900 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:1073741824
:3:rocvirtual.cpp           :798 : 4815821498908 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = val:
:3:rocvirtual.cpp           :723 : 4815821498913 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x705265a00000 obj:[0x705165a00000-0x705365a00000]
:3:rocvirtual.cpp           :3016: 4815821498918 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIN3c104HalfEEENS_6detail5ArrayIPcLi1EEEEEviT0_T1_
:3:hip_module.cpp           :670 : 4815821498927 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821498932 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821498940 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2ecc )
:3:hip_device_runtime.cpp   :631 : 4815821498945 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821498954 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {1048576,1,1}, {256,1,1}, 0, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821498960 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821498968 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {1048576,1,1}, {256,1,1}, 0x70536bbc2f80, 0x70536bbc2f70 )
:3:hip_platform.cpp         :237 : 4815821498973 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821498983 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a988e9d0, {1048576,1,1}, {256,1,1}, 0x70536bbc2fa0, 0, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821498990 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:1073741824
:3:rocvirtual.cpp           :798 : 4815821498996 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = val:
:3:rocvirtual.cpp           :723 : 4815821499001 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x7052e5a00000 obj:[0x705165a00000-0x705365a00000]
:3:rocvirtual.cpp           :3016: 4815821499007 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIN3c104HalfEEENS_6detail5ArrayIPcLi1EEEEEviT0_T1_
:3:hip_module.cpp           :670 : 4815821499016 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821499021 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :653 : 4815821499030 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821499035 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821499040 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821499046 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499059 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3684 )
:3:hip_device_runtime.cpp   :631 : 4815821499073 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499081 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc35b4 )
:3:hip_device_runtime.cpp   :631 : 4815821499087 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499094 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc340c )
:3:hip_device_runtime.cpp   :631 : 4815821499098 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821499108 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821499113 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821499119 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821499126 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499135 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3684 )
:3:hip_device_runtime.cpp   :631 : 4815821499140 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499147 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc35b4 )
:3:hip_device_runtime.cpp   :631 : 4815821499154 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499160 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc340c )
:3:hip_device_runtime.cpp   :631 : 4815821499166 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821499173 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821499179 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821499185 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821499191 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499198 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3684 )
:3:hip_device_runtime.cpp   :631 : 4815821499204 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499211 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc35b4 )
:3:hip_device_runtime.cpp   :631 : 4815821499216 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499222 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc340c )
:3:hip_device_runtime.cpp   :631 : 4815821499228 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821499236 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821499241 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821499247 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821499275 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499295 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3b4c )
:3:hip_device_runtime.cpp   :631 : 4815821499302 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821499350 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc33e8, 3, 0x705380800000 )
:3:hip_memory.cpp           :3866: 4815821499358 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821499367 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc33e8, 3, 0x705384a00000 )
:3:hip_memory.cpp           :3866: 4815821499372 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821499380 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc33e8, 3, 0x7053aca00000 )
:3:hip_memory.cpp           :3866: 4815821499386 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_module.cpp           :438 : 4815821499399 us: [pid:2429112 tid:0x70536bbc5640]  hipModuleLaunchKernel ( 0x0x7053a00bdb10, 8192, 1, 1, 256, 1, 1, 256, stream:<null>, 0x70536bbc33f0, char array:<null> )
:3:rocvirtual.cpp           :723 : 4815821499407 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = ptr:0x705380800000 obj:[0x705380800000-0x705384800000]
:3:rocvirtual.cpp           :723 : 4815821499414 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = ptr:0x705384a00000 obj:[0x705384a00000-0x705388a00000]
:3:rocvirtual.cpp           :723 : 4815821499420 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x7053aca00000 obj:[0x7053ac800000-0x7053adc00000]
:3:rocvirtual.cpp           :3016: 4815821499426 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _bwd_preprocess
:3:hip_module.cpp           :458 : 4815821499434 us: [pid:2429112 tid:0x70536bbc5640] hipModuleLaunchKernel: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499461 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3b4c )
:3:hip_device_runtime.cpp   :631 : 4815821499467 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821499502 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3b4c )
:3:hip_device_runtime.cpp   :631 : 4815821499514 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device.cpp           :477 : 4815821499520 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevicePropertiesR0000 ( 0x70536bbc3470, 0 )
:3:hip_device.cpp           :578 : 4815821499525 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevicePropertiesR0000: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821509033 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3b4c )
:3:hip_device_runtime.cpp   :631 : 4815821509050 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device.cpp           :477 : 4815821509692 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevicePropertiesR0000 ( 0x70536bbc3470, 0 )
:3:hip_device.cpp           :578 : 4815821509703 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevicePropertiesR0000: Returned hipSuccess :
:3:hip_module.cpp           :65  : 4815821509721 us: [pid:2429112 tid:0x70536bbc5640]  hipModuleLoadDataEx ( 0x70536bbc3720, 0x7053a00c9380 )
:3:devprogram.cpp           :2676: 4815821509962 us: [pid:2429112 tid:0x70536bbc5640] Using Code Object V4.
:3:hip_module.cpp           :66  : 4815821604298 us: [pid:2429112 tid:0x70536bbc5640] hipModuleLoadDataEx: Returned hipSuccess :
:3:hip_module.cpp           :74  : 4815821604311 us: [pid:2429112 tid:0x70536bbc5640]  hipModuleGetFunction ( 0x70536bbc3728, 0x94e2e00, _bwd_kernel )
:3:hip_module.cpp           :88  : 4815821604317 us: [pid:2429112 tid:0x70536bbc5640] hipModuleGetFunction: Returned hipSuccess :
:3:hip_module.cpp           :116 : 4815821604323 us: [pid:2429112 tid:0x70536bbc5640]  hipFuncGetAttribute ( 0x70536bbc3700, HIP_FUNC_ATTRIBUTE_NUM_REGS, 0x0x7053a00d9cd0 )
:3:hip_module.cpp           :173 : 4815821604328 us: [pid:2429112 tid:0x70536bbc5640] hipFuncGetAttribute: Returned hipSuccess :
:3:hip_module.cpp           :116 : 4815821604335 us: [pid:2429112 tid:0x70536bbc5640]  hipFuncGetAttribute ( 0x70536bbc3704, HIP_FUNC_ATTRIBUTE_LOCAL_SIZE_BYTES, 0x0x7053a00d9cd0 )
:3:hip_module.cpp           :173 : 4815821604338 us: [pid:2429112 tid:0x70536bbc5640] hipFuncGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821604366 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705394e00000 )
:3:hip_memory.cpp           :3866: 4815821604376 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821604382 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x70538ce00000 )
:3:hip_memory.cpp           :3866: 4815821604388 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821604393 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705388c00000 )
:3:hip_memory.cpp           :3866: 4815821604399 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821604405 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705380800000 )
:3:hip_memory.cpp           :3866: 4815821604411 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821604418 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705384a00000 )
:3:hip_memory.cpp           :3866: 4815821604422 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821604428 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705165a00000 )
:3:hip_memory.cpp           :3866: 4815821604433 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821604439 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705374200000 )
:3:hip_memory.cpp           :3866: 4815821604444 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821604450 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705378400000 )
:3:hip_memory.cpp           :3866: 4815821604455 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821604461 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x7053ac800000 )
:3:hip_memory.cpp           :3866: 4815821604465 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821604471 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x7053aca00000 )
:3:hip_memory.cpp           :3866: 4815821604476 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_module.cpp           :438 : 4815821604488 us: [pid:2429112 tid:0x70536bbc5640]  hipModuleLaunchKernel ( 0x0x7053a00d9cd0, 64, 128, 1, 512, 1, 1, 16384, stream:<null>, 0x70536bbc3340, char array:<null> )
:3:rocvirtual.cpp           :723 : 4815821604503 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = ptr:0x705394e00000 obj:[0x705394e00000-0x705398e00000]
:3:rocvirtual.cpp           :723 : 4815821604509 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = ptr:0x70538ce00000 obj:[0x70538ce00000-0x705390e00000]
:3:rocvirtual.cpp           :723 : 4815821604512 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x705388c00000 obj:[0x705388c00000-0x70538cc00000]
:3:rocvirtual.cpp           :798 : 4815821604518 us: [pid:2429112 tid:0x70536bbc5640] Arg3:   = val:1040187392
:3:rocvirtual.cpp           :723 : 4815821604523 us: [pid:2429112 tid:0x70536bbc5640] Arg4:   = ptr:0x705380800000 obj:[0x705380800000-0x705384800000]
:3:rocvirtual.cpp           :723 : 4815821604527 us: [pid:2429112 tid:0x70536bbc5640] Arg5:   = ptr:0x705384a00000 obj:[0x705384a00000-0x705388a00000]
:3:rocvirtual.cpp           :723 : 4815821604532 us: [pid:2429112 tid:0x70536bbc5640] Arg6:   = ptr:0x705165a00000 obj:[0x705165a00000-0x705365a00000]
:3:rocvirtual.cpp           :723 : 4815821604537 us: [pid:2429112 tid:0x70536bbc5640] Arg7:   = ptr:0x705374200000 obj:[0x705374200000-0x705378200000]
:3:rocvirtual.cpp           :723 : 4815821604540 us: [pid:2429112 tid:0x70536bbc5640] Arg8:   = ptr:0x705378400000 obj:[0x705378400000-0x70537c400000]
:3:rocvirtual.cpp           :723 : 4815821604545 us: [pid:2429112 tid:0x70536bbc5640] Arg9:   = ptr:0x7053ac800000 obj:[0x7053ac800000-0x7053adc00000]
:3:rocvirtual.cpp           :723 : 4815821604549 us: [pid:2429112 tid:0x70536bbc5640] Arg10:   = ptr:0x7053aca00000 obj:[0x7053ac800000-0x7053adc00000]
:3:rocvirtual.cpp           :798 : 4815821604553 us: [pid:2429112 tid:0x70536bbc5640] Arg11:   = val:33554432
:3:rocvirtual.cpp           :798 : 4815821604557 us: [pid:2429112 tid:0x70536bbc5640] Arg12:   = val:16777216
:3:rocvirtual.cpp           :798 : 4815821604560 us: [pid:2429112 tid:0x70536bbc5640] Arg13:   = val:524288
:3:rocvirtual.cpp           :798 : 4815821604563 us: [pid:2429112 tid:0x70536bbc5640] Arg14:   = val:64
:3:rocvirtual.cpp           :798 : 4815821604567 us: [pid:2429112 tid:0x70536bbc5640] Arg15:   = val:16777216
:3:rocvirtual.cpp           :798 : 4815821604572 us: [pid:2429112 tid:0x70536bbc5640] Arg16:   = val:524288
:3:rocvirtual.cpp           :798 : 4815821604575 us: [pid:2429112 tid:0x70536bbc5640] Arg17:   = val:64
:3:rocvirtual.cpp           :798 : 4815821604578 us: [pid:2429112 tid:0x70536bbc5640] Arg18:   = val:16777216
:3:rocvirtual.cpp           :798 : 4815821604582 us: [pid:2429112 tid:0x70536bbc5640] Arg19:   = val:524288
:3:rocvirtual.cpp           :798 : 4815821604586 us: [pid:2429112 tid:0x70536bbc5640] Arg20:   = val:64
:3:rocvirtual.cpp           :798 : 4815821604590 us: [pid:2429112 tid:0x70536bbc5640] Arg21:   = val:2
:3:rocvirtual.cpp           :798 : 4815821604594 us: [pid:2429112 tid:0x70536bbc5640] Arg22:   = val:32
:3:rocvirtual.cpp           :798 : 4815821604599 us: [pid:2429112 tid:0x70536bbc5640] Arg23:   = val:8192
:3:rocvirtual.cpp           :798 : 4815821604604 us: [pid:2429112 tid:0x70536bbc5640] Arg24:   = val:524288
:3:rocvirtual.cpp           :798 : 4815821604607 us: [pid:2429112 tid:0x70536bbc5640] Arg25:   = val:67108864
:3:rocvirtual.cpp           :3016: 4815821604612 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _bwd_kernel
:3:hip_module.cpp           :458 : 4815821604621 us: [pid:2429112 tid:0x70536bbc5640] hipModuleLaunchKernel: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821604751 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc352c )
:3:hip_device_runtime.cpp   :631 : 4815821604757 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821604767 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3414 )
:3:hip_device_runtime.cpp   :631 : 4815821604772 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821604781 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc325c )
:3:hip_device_runtime.cpp   :631 : 4815821604787 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821604804 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821604809 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821604844 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2dbc )
:3:hip_device_runtime.cpp   :631 : 4815821604849 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821604860 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821604865 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc28cc )
:3:hip_device_runtime.cpp   :631 : 4815821604869 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_graph.cpp            :826 : 4815821604875 us: [pid:2429112 tid:0x70536bbc5640]  hipStreamIsCapturing ( stream:<null>, 0x70536bbc2960 )
:3:hip_graph.cpp            :827 : 4815821604881 us: [pid:2429112 tid:0x70536bbc5640] hipStreamIsCapturing: Returned hipSuccess :
:3:hip_memory.cpp           :599 : 4815821604890 us: [pid:2429112 tid:0x70536bbc5640]  hipMalloc ( 0x70536bbc2a08, 134217728 )
:3:rocdevice.cpp            :2349: 4815821604975 us: [pid:2429112 tid:0x70536bbc5640] device=0x6fee8c0, freeMem_ = 0xdd5c00000
:3:hip_memory.cpp           :601 : 4815821604985 us: [pid:2429112 tid:0x70536bbc5640] hipMalloc: Returned hipSuccess : 0x70515d800000: duration: 95 us
:3:hip_device_runtime.cpp   :623 : 4815821605000 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27cc )
:3:hip_device_runtime.cpp   :631 : 4815821605004 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821605010 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821605014 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821605018 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821605023 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821605028 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc24dc )
:3:hip_device_runtime.cpp   :631 : 4815821605033 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821605041 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {131072,1,1}, {64,2,1}, 2048, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821605047 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821605054 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {4294967152,4294967295,4126967744}, {1807492488,28755,1807492536}, 0x70536bbc2560, 0x70536bbc2558 )
:3:hip_platform.cpp         :237 : 4815821605059 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821605066 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a989b558, {131072,1,1}, {64,2,1}, 0x70536bbc2550, 2048, stream:<null> )
:3:devprogram.cpp           :2679: 4815821606753 us: [pid:2429112 tid:0x70536bbc5640] Using Code Object V5.
:3:rocvirtual.cpp           :798 : 4815821609244 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:
:3:rocvirtual.cpp           :3016: 4815821609254 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native13reduce_kernelILi128ELi4ENS0_8ReduceOpIN3c104HalfENS0_14func_wrapper_tIS4_ZNS0_11sum_functorIS4_fS4_EclERNS_14TensorIteratorEEUlffE_EEjS4_Li4EEEEEvT1_
:3:hip_module.cpp           :670 : 4815821609260 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821609266 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821609274 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27cc )
:3:hip_device_runtime.cpp   :631 : 4815821609279 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609282 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821609286 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609291 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821609295 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609300 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc24dc )
:3:hip_device_runtime.cpp   :631 : 4815821609305 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821609312 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {131072,1,1}, {64,2,1}, 2048, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821609317 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821609324 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {131072,1,1}, {64,2,1}, 0x70536bbc2560, 0x70536bbc2558 )
:3:hip_platform.cpp         :237 : 4815821609328 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821609335 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a989b558, {131072,1,1}, {64,2,1}, 0x70536bbc2550, 2048, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821609340 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:
:3:rocvirtual.cpp           :3016: 4815821609344 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native13reduce_kernelILi128ELi4ENS0_8ReduceOpIN3c104HalfENS0_14func_wrapper_tIS4_ZNS0_11sum_functorIS4_fS4_EclERNS_14TensorIteratorEEUlffE_EEjS4_Li4EEEEEvT1_
:3:hip_module.cpp           :670 : 4815821609350 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821609354 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821609362 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27cc )
:3:hip_device_runtime.cpp   :631 : 4815821609365 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609370 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821609373 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609377 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821609380 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609385 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc24dc )
:3:hip_device_runtime.cpp   :631 : 4815821609389 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821609394 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {131072,1,1}, {64,2,1}, 2048, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821609397 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821609403 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {131072,1,1}, {64,2,1}, 0x70536bbc2560, 0x70536bbc2558 )
:3:hip_platform.cpp         :237 : 4815821609407 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821609414 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a989b558, {131072,1,1}, {64,2,1}, 0x70536bbc2550, 2048, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821609418 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:
:3:rocvirtual.cpp           :3016: 4815821609422 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native13reduce_kernelILi128ELi4ENS0_8ReduceOpIN3c104HalfENS0_14func_wrapper_tIS4_ZNS0_11sum_functorIS4_fS4_EclERNS_14TensorIteratorEEUlffE_EEjS4_Li4EEEEEvT1_
:3:hip_module.cpp           :670 : 4815821609427 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821609431 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821609435 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27cc )
:3:hip_device_runtime.cpp   :631 : 4815821609440 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609445 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821609449 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609453 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821609457 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609461 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc24dc )
:3:hip_device_runtime.cpp   :631 : 4815821609464 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821609469 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {131072,1,1}, {64,2,1}, 2048, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821609473 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821609478 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {131072,1,1}, {64,2,1}, 0x70536bbc2560, 0x70536bbc2558 )
:3:hip_platform.cpp         :237 : 4815821609482 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821609488 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a989b558, {131072,1,1}, {64,2,1}, 0x70536bbc2550, 2048, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821609493 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:
:3:rocvirtual.cpp           :3016: 4815821609496 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native13reduce_kernelILi128ELi4ENS0_8ReduceOpIN3c104HalfENS0_14func_wrapper_tIS4_ZNS0_11sum_functorIS4_fS4_EclERNS_14TensorIteratorEEUlffE_EEjS4_Li4EEEEEvT1_
:3:hip_module.cpp           :670 : 4815821609501 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821609505 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :653 : 4815821609514 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821609519 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821609560 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821609565 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :638 : 4815821609590 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDeviceCount ( 0x70536bbc4804 )
:3:hip_device_runtime.cpp   :640 : 4815821609596 us: [pid:2429112 tid:0x70536bbc5640] hipGetDeviceCount: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821609600 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821609604 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609609 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc47d8 )
:3:hip_device_runtime.cpp   :631 : 4815821609614 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609627 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc44b4 )
:3:hip_device_runtime.cpp   :631 : 4815821609631 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821609655 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821609660 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :638 : 4815821609666 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDeviceCount ( 0x70536bbc4804 )
:3:hip_device_runtime.cpp   :640 : 4815821609671 us: [pid:2429112 tid:0x70536bbc5640] hipGetDeviceCount: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821609675 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821609680 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609684 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc47d8 )
:3:hip_device_runtime.cpp   :631 : 4815821609687 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609693 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc44b4 )
:3:hip_device_runtime.cpp   :631 : 4815821609697 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821609705 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821609710 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :638 : 4815821609715 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDeviceCount ( 0x70536bbc4804 )
:3:hip_device_runtime.cpp   :640 : 4815821609719 us: [pid:2429112 tid:0x70536bbc5640] hipGetDeviceCount: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821609723 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821609728 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609733 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc47d8 )
:3:hip_device_runtime.cpp   :631 : 4815821609737 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609742 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc44b4 )
:3:hip_device_runtime.cpp   :631 : 4815821609747 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821609753 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821609757 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :638 : 4815821609846 us: [pid:2429112 tid:0x7056c923d440]  hipGetDeviceCount ( 0x7fffacce4064 )
:3:hip_device_runtime.cpp   :640 : 4815821609857 us: [pid:2429112 tid:0x7056c923d440] hipGetDeviceCount: Returned hipSuccess :
:3:hip_context.cpp          :340 : 4815821609865 us: [pid:2429112 tid:0x7056c923d440]  hipDevicePrimaryCtxGetState ( 0, 0x7fffacce3eb8, 0x7fffacce3ebc )
:3:hip_context.cpp          :354 : 4815821609871 us: [pid:2429112 tid:0x7056c923d440] hipDevicePrimaryCtxGetState: Returned hipSuccess :
:3:hip_device_runtime.cpp   :638 : 4815821609889 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDeviceCount ( 0x70536bbc4804 )
:3:hip_device_runtime.cpp   :640 : 4815821609895 us: [pid:2429112 tid:0x70536bbc5640] hipGetDeviceCount: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821609900 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821609904 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609909 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc47d8 )
:3:hip_device_runtime.cpp   :631 : 4815821609913 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821609919 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc44b4 )
:3:hip_device_runtime.cpp   :631 : 4815821609923 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :

[4] failed call to backward
INFO:triton.ops.flash_attention:backward
:3:hip_device_runtime.cpp   :623 : 4815821610051 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3e7c )
:3:hip_device_runtime.cpp   :631 : 4815821610061 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610087 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3e7c )
:3:hip_device_runtime.cpp   :631 : 4815821610093 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device.cpp           :477 : 4815821610101 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevicePropertiesR0000 ( 0x70536bbc37a0, 0 )
:3:hip_device.cpp           :578 : 4815821610107 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevicePropertiesR0000: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610159 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc36d4 )
:3:hip_device_runtime.cpp   :631 : 4815821610165 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610173 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3624 )
:3:hip_device_runtime.cpp   :631 : 4815821610178 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610186 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc346c )
:3:hip_device_runtime.cpp   :631 : 4815821610193 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821610204 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821610211 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821610218 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821610224 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610233 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3914 )
:3:hip_device_runtime.cpp   :631 : 4815821610239 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610248 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc37b4 )
:3:hip_device_runtime.cpp   :631 : 4815821610255 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610268 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2ecc )
:3:hip_device_runtime.cpp   :631 : 4815821610274 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821610281 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {1048576,1,1}, {256,1,1}, 0, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821610285 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821610292 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {4294967152,4294967295,0}, {4126967784,28757,1807495048}, 0x70536bbc2f80, 0x70536bbc2f70 )
:3:hip_platform.cpp         :237 : 4815821610296 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821610303 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a988e9d0, {1048576,1,1}, {256,1,1}, 0x70536bbc2fa0, 0, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821610314 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:1073741824
:3:rocvirtual.cpp           :798 : 4815821610318 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = val:
:3:rocvirtual.cpp           :723 : 4815821610323 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x705165a00000 obj:[0x705165a00000-0x705365a00000]
:3:rocvirtual.cpp           :3016: 4815821610327 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIN3c104HalfEEENS_6detail5ArrayIPcLi1EEEEEviT0_T1_
:3:hip_module.cpp           :670 : 4815821610333 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821610337 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821610342 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2ecc )
:3:hip_device_runtime.cpp   :631 : 4815821610345 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821610352 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {1048576,1,1}, {256,1,1}, 0, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821610357 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821610362 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {1048576,1,1}, {256,1,1}, 0x70536bbc2f80, 0x70536bbc2f70 )
:3:hip_platform.cpp         :237 : 4815821610368 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821610375 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a988e9d0, {1048576,1,1}, {256,1,1}, 0x70536bbc2fa0, 0, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821610380 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:1073741824
:3:rocvirtual.cpp           :798 : 4815821610383 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = val:
:3:rocvirtual.cpp           :723 : 4815821610388 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x7051e5a00000 obj:[0x705165a00000-0x705365a00000]
:3:rocvirtual.cpp           :3016: 4815821610391 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIN3c104HalfEEENS_6detail5ArrayIPcLi1EEEEEviT0_T1_
:3:hip_module.cpp           :670 : 4815821610396 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821610402 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821610407 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2ecc )
:3:hip_device_runtime.cpp   :631 : 4815821610411 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821610417 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {1048576,1,1}, {256,1,1}, 0, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821610421 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821610426 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {1048576,1,1}, {256,1,1}, 0x70536bbc2f80, 0x70536bbc2f70 )
:3:hip_platform.cpp         :237 : 4815821610430 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821610437 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a988e9d0, {1048576,1,1}, {256,1,1}, 0x70536bbc2fa0, 0, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821610442 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:1073741824
:3:rocvirtual.cpp           :798 : 4815821610445 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = val:
:3:rocvirtual.cpp           :723 : 4815821610450 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x705265a00000 obj:[0x705165a00000-0x705365a00000]
:3:rocvirtual.cpp           :3016: 4815821610454 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIN3c104HalfEEENS_6detail5ArrayIPcLi1EEEEEviT0_T1_
:3:hip_module.cpp           :670 : 4815821610459 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821610464 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821610470 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2ecc )
:3:hip_device_runtime.cpp   :631 : 4815821610473 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821610479 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {1048576,1,1}, {256,1,1}, 0, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821610483 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821610490 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {1048576,1,1}, {256,1,1}, 0x70536bbc2f80, 0x70536bbc2f70 )
:3:hip_platform.cpp         :237 : 4815821610495 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821610502 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a988e9d0, {1048576,1,1}, {256,1,1}, 0x70536bbc2fa0, 0, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821610507 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:1073741824
:3:rocvirtual.cpp           :798 : 4815821610510 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = val:
:3:rocvirtual.cpp           :723 : 4815821610514 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x7052e5a00000 obj:[0x705165a00000-0x705365a00000]
:3:rocvirtual.cpp           :3016: 4815821610517 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIN3c104HalfEEENS_6detail5ArrayIPcLi1EEEEEviT0_T1_
:3:hip_module.cpp           :670 : 4815821610522 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821610527 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :653 : 4815821610532 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821610536 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821610541 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821610545 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610560 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3684 )
:3:hip_device_runtime.cpp   :631 : 4815821610564 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610569 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc35b4 )
:3:hip_device_runtime.cpp   :631 : 4815821610573 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610579 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc340c )
:3:hip_device_runtime.cpp   :631 : 4815821610583 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821610589 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821610594 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821610598 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821610601 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610607 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3684 )
:3:hip_device_runtime.cpp   :631 : 4815821610611 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610615 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc35b4 )
:3:hip_device_runtime.cpp   :631 : 4815821610618 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610622 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc340c )
:3:hip_device_runtime.cpp   :631 : 4815821610626 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821610631 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821610635 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821610639 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821610642 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610648 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3684 )
:3:hip_device_runtime.cpp   :631 : 4815821610652 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610657 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc35b4 )
:3:hip_device_runtime.cpp   :631 : 4815821610662 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610666 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc340c )
:3:hip_device_runtime.cpp   :631 : 4815821610670 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821610677 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821610682 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821610687 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821610691 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610704 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3b4c )
:3:hip_device_runtime.cpp   :631 : 4815821610709 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610745 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc33e8, 3, 0x705380800000 )
:3:hip_memory.cpp           :3866: 4815821610751 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610758 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc33e8, 3, 0x705384a00000 )
:3:hip_memory.cpp           :3866: 4815821610763 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610769 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc33e8, 3, 0x7053aca00000 )
:3:hip_memory.cpp           :3866: 4815821610773 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_module.cpp           :438 : 4815821610783 us: [pid:2429112 tid:0x70536bbc5640]  hipModuleLaunchKernel ( 0x0x7053a00bdb10, 8192, 1, 1, 256, 1, 1, 256, stream:<null>, 0x70536bbc33f0, char array:<null> )
:3:rocvirtual.cpp           :723 : 4815821610789 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = ptr:0x705380800000 obj:[0x705380800000-0x705384800000]
:3:rocvirtual.cpp           :723 : 4815821610793 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = ptr:0x705384a00000 obj:[0x705384a00000-0x705388a00000]
:3:rocvirtual.cpp           :723 : 4815821610798 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x7053aca00000 obj:[0x7053ac800000-0x7053adc00000]
:3:rocvirtual.cpp           :3016: 4815821610802 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _bwd_preprocess
:3:hip_module.cpp           :458 : 4815821610810 us: [pid:2429112 tid:0x70536bbc5640] hipModuleLaunchKernel: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821610832 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3b4c )
:3:hip_device_runtime.cpp   :631 : 4815821610838 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610875 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705394e00000 )
:3:hip_memory.cpp           :3866: 4815821610881 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610887 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x70538ce00000 )
:3:hip_memory.cpp           :3866: 4815821610892 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610898 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705388c00000 )
:3:hip_memory.cpp           :3866: 4815821610902 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610907 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705380800000 )
:3:hip_memory.cpp           :3866: 4815821610912 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610918 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705384a00000 )
:3:hip_memory.cpp           :3866: 4815821610922 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610927 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705165a00000 )
:3:hip_memory.cpp           :3866: 4815821610932 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610937 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705374200000 )
:3:hip_memory.cpp           :3866: 4815821610941 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610946 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x705378400000 )
:3:hip_memory.cpp           :3866: 4815821610949 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610955 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x7053ac800000 )
:3:hip_memory.cpp           :3866: 4815821610959 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_memory.cpp           :3860: 4815821610965 us: [pid:2429112 tid:0x70536bbc5640]  hipPointerGetAttribute ( 0x70536bbc32e8, 3, 0x7053aca00000 )
:3:hip_memory.cpp           :3866: 4815821610969 us: [pid:2429112 tid:0x70536bbc5640] hipPointerGetAttribute: Returned hipSuccess :
:3:hip_module.cpp           :438 : 4815821610977 us: [pid:2429112 tid:0x70536bbc5640]  hipModuleLaunchKernel ( 0x0x7053a00d9cd0, 64, 128, 1, 512, 1, 1, 16384, stream:<null>, 0x70536bbc3340, char array:<null> )
:3:rocvirtual.cpp           :723 : 4815821610984 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = ptr:0x705394e00000 obj:[0x705394e00000-0x705398e00000]
:3:rocvirtual.cpp           :723 : 4815821610988 us: [pid:2429112 tid:0x70536bbc5640] Arg1:   = ptr:0x70538ce00000 obj:[0x70538ce00000-0x705390e00000]
:3:rocvirtual.cpp           :723 : 4815821610991 us: [pid:2429112 tid:0x70536bbc5640] Arg2:   = ptr:0x705388c00000 obj:[0x705388c00000-0x70538cc00000]
:3:rocvirtual.cpp           :798 : 4815821610995 us: [pid:2429112 tid:0x70536bbc5640] Arg3:   = val:1040187392
:3:rocvirtual.cpp           :723 : 4815821610999 us: [pid:2429112 tid:0x70536bbc5640] Arg4:   = ptr:0x705380800000 obj:[0x705380800000-0x705384800000]
:3:rocvirtual.cpp           :723 : 4815821611003 us: [pid:2429112 tid:0x70536bbc5640] Arg5:   = ptr:0x705384a00000 obj:[0x705384a00000-0x705388a00000]
:3:rocvirtual.cpp           :723 : 4815821611007 us: [pid:2429112 tid:0x70536bbc5640] Arg6:   = ptr:0x705165a00000 obj:[0x705165a00000-0x705365a00000]
:3:rocvirtual.cpp           :723 : 4815821611011 us: [pid:2429112 tid:0x70536bbc5640] Arg7:   = ptr:0x705374200000 obj:[0x705374200000-0x705378200000]
:3:rocvirtual.cpp           :723 : 4815821611017 us: [pid:2429112 tid:0x70536bbc5640] Arg8:   = ptr:0x705378400000 obj:[0x705378400000-0x70537c400000]
:3:rocvirtual.cpp           :723 : 4815821611021 us: [pid:2429112 tid:0x70536bbc5640] Arg9:   = ptr:0x7053ac800000 obj:[0x7053ac800000-0x7053adc00000]
:3:rocvirtual.cpp           :723 : 4815821611026 us: [pid:2429112 tid:0x70536bbc5640] Arg10:   = ptr:0x7053aca00000 obj:[0x7053ac800000-0x7053adc00000]
:3:rocvirtual.cpp           :798 : 4815821611031 us: [pid:2429112 tid:0x70536bbc5640] Arg11:   = val:33554432
:3:rocvirtual.cpp           :798 : 4815821611035 us: [pid:2429112 tid:0x70536bbc5640] Arg12:   = val:16777216
:3:rocvirtual.cpp           :798 : 4815821611039 us: [pid:2429112 tid:0x70536bbc5640] Arg13:   = val:524288
:3:rocvirtual.cpp           :798 : 4815821611043 us: [pid:2429112 tid:0x70536bbc5640] Arg14:   = val:64
:3:rocvirtual.cpp           :798 : 4815821611048 us: [pid:2429112 tid:0x70536bbc5640] Arg15:   = val:16777216
:3:rocvirtual.cpp           :798 : 4815821611053 us: [pid:2429112 tid:0x70536bbc5640] Arg16:   = val:524288
:3:rocvirtual.cpp           :798 : 4815821611057 us: [pid:2429112 tid:0x70536bbc5640] Arg17:   = val:64
:3:rocvirtual.cpp           :798 : 4815821611061 us: [pid:2429112 tid:0x70536bbc5640] Arg18:   = val:16777216
:3:rocvirtual.cpp           :798 : 4815821611066 us: [pid:2429112 tid:0x70536bbc5640] Arg19:   = val:524288
:3:rocvirtual.cpp           :798 : 4815821611071 us: [pid:2429112 tid:0x70536bbc5640] Arg20:   = val:64
:3:rocvirtual.cpp           :798 : 4815821611076 us: [pid:2429112 tid:0x70536bbc5640] Arg21:   = val:2
:3:rocvirtual.cpp           :798 : 4815821611079 us: [pid:2429112 tid:0x70536bbc5640] Arg22:   = val:32
:3:rocvirtual.cpp           :798 : 4815821611084 us: [pid:2429112 tid:0x70536bbc5640] Arg23:   = val:8192
:3:rocvirtual.cpp           :798 : 4815821611088 us: [pid:2429112 tid:0x70536bbc5640] Arg24:   = val:524288
:3:rocvirtual.cpp           :798 : 4815821611093 us: [pid:2429112 tid:0x70536bbc5640] Arg25:   = val:67108864
:3:rocvirtual.cpp           :3016: 4815821611096 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _bwd_kernel
:3:hip_module.cpp           :458 : 4815821611101 us: [pid:2429112 tid:0x70536bbc5640] hipModuleLaunchKernel: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611114 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc352c )
:3:hip_device_runtime.cpp   :631 : 4815821611120 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611124 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc3414 )
:3:hip_device_runtime.cpp   :631 : 4815821611128 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611133 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc325c )
:3:hip_device_runtime.cpp   :631 : 4815821611138 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821611145 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821611149 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611160 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc2dbc )
:3:hip_device_runtime.cpp   :631 : 4815821611164 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611171 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27cc )
:3:hip_device_runtime.cpp   :631 : 4815821611176 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611181 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821611184 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611190 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821611194 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611201 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc24dc )
:3:hip_device_runtime.cpp   :631 : 4815821611204 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821611213 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {131072,1,1}, {64,2,1}, 2048, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821611218 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821611225 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {4294967152,4294967295,4126967744}, {1807492488,28755,1807492536}, 0x70536bbc2560, 0x70536bbc2558 )
:3:hip_platform.cpp         :237 : 4815821611228 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821611235 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a989b558, {131072,1,1}, {64,2,1}, 0x70536bbc2550, 2048, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821611242 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:
:3:rocvirtual.cpp           :3016: 4815821611247 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native13reduce_kernelILi128ELi4ENS0_8ReduceOpIN3c104HalfENS0_14func_wrapper_tIS4_ZNS0_11sum_functorIS4_fS4_EclERNS_14TensorIteratorEEUlffE_EEjS4_Li4EEEEEvT1_
:3:hip_module.cpp           :670 : 4815821611254 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821611258 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821611265 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27cc )
:3:hip_device_runtime.cpp   :631 : 4815821611269 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611274 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821611277 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611282 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821611286 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611291 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc24dc )
:3:hip_device_runtime.cpp   :631 : 4815821611295 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821611301 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {131072,1,1}, {64,2,1}, 2048, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821611305 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821611311 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {131072,1,1}, {64,2,1}, 0x70536bbc2560, 0x70536bbc2558 )
:3:hip_platform.cpp         :237 : 4815821611315 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821611322 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a989b558, {131072,1,1}, {64,2,1}, 0x70536bbc2550, 2048, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821611326 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:
:3:rocvirtual.cpp           :3016: 4815821611329 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native13reduce_kernelILi128ELi4ENS0_8ReduceOpIN3c104HalfENS0_14func_wrapper_tIS4_ZNS0_11sum_functorIS4_fS4_EclERNS_14TensorIteratorEEUlffE_EEjS4_Li4EEEEEvT1_
:3:hip_module.cpp           :670 : 4815821611334 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821611338 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821611342 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27cc )
:3:hip_device_runtime.cpp   :631 : 4815821611346 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611349 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821611353 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611357 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821611361 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611365 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc24dc )
:3:hip_device_runtime.cpp   :631 : 4815821611369 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821611375 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {131072,1,1}, {64,2,1}, 2048, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821611382 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821611388 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {131072,1,1}, {64,2,1}, 0x70536bbc2560, 0x70536bbc2558 )
:3:hip_platform.cpp         :237 : 4815821611392 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821611399 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a989b558, {131072,1,1}, {64,2,1}, 0x70536bbc2550, 2048, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821611405 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:
:3:rocvirtual.cpp           :3016: 4815821611410 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native13reduce_kernelILi128ELi4ENS0_8ReduceOpIN3c104HalfENS0_14func_wrapper_tIS4_ZNS0_11sum_functorIS4_fS4_EclERNS_14TensorIteratorEEUlffE_EEjS4_Li4EEEEEvT1_
:3:hip_module.cpp           :670 : 4815821611416 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821611421 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :623 : 4815821611425 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27cc )
:3:hip_device_runtime.cpp   :631 : 4815821611429 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611434 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821611437 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611441 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc27dc )
:3:hip_device_runtime.cpp   :631 : 4815821611445 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611450 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc24dc )
:3:hip_device_runtime.cpp   :631 : 4815821611454 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_platform.cpp         :219 : 4815821611459 us: [pid:2429112 tid:0x70536bbc5640]  __hipPushCallConfiguration ( {131072,1,1}, {64,2,1}, 2048, stream:<null> )
:3:hip_platform.cpp         :223 : 4815821611463 us: [pid:2429112 tid:0x70536bbc5640] __hipPushCallConfiguration: Returned hipSuccess :
:3:hip_platform.cpp         :228 : 4815821611469 us: [pid:2429112 tid:0x70536bbc5640]  __hipPopCallConfiguration ( {131072,1,1}, {64,2,1}, 0x70536bbc2560, 0x70536bbc2558 )
:3:hip_platform.cpp         :237 : 4815821611475 us: [pid:2429112 tid:0x70536bbc5640] __hipPopCallConfiguration: Returned hipSuccess :
:3:hip_module.cpp           :669 : 4815821611481 us: [pid:2429112 tid:0x70536bbc5640]  hipLaunchKernel ( 0x7056a989b558, {131072,1,1}, {64,2,1}, 0x70536bbc2550, 2048, stream:<null> )
:3:rocvirtual.cpp           :798 : 4815821611487 us: [pid:2429112 tid:0x70536bbc5640] Arg0:   = val:
:3:rocvirtual.cpp           :3016: 4815821611491 us: [pid:2429112 tid:0x70536bbc5640] ShaderName : _ZN2at6native13reduce_kernelILi128ELi4ENS0_8ReduceOpIN3c104HalfENS0_14func_wrapper_tIS4_ZNS0_11sum_functorIS4_fS4_EclERNS_14TensorIteratorEEUlffE_EEjS4_Li4EEEEEvT1_
:3:hip_module.cpp           :670 : 4815821611497 us: [pid:2429112 tid:0x70536bbc5640] hipLaunchKernel: Returned hipSuccess :
:3:hip_error.cpp            :36  : 4815821611502 us: [pid:2429112 tid:0x70536bbc5640]  hipGetLastError (  )
:3:hip_device_runtime.cpp   :653 : 4815821611509 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821611512 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821611531 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821611535 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :638 : 4815821611544 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDeviceCount ( 0x70536bbc4804 )
:3:hip_device_runtime.cpp   :640 : 4815821611550 us: [pid:2429112 tid:0x70536bbc5640] hipGetDeviceCount: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821611555 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821611559 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611564 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc47d8 )
:3:hip_device_runtime.cpp   :631 : 4815821611570 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611577 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc44b4 )
:3:hip_device_runtime.cpp   :631 : 4815821611581 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821611594 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821611599 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :638 : 4815821611605 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDeviceCount ( 0x70536bbc4804 )
:3:hip_device_runtime.cpp   :640 : 4815821611608 us: [pid:2429112 tid:0x70536bbc5640] hipGetDeviceCount: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821611612 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821611617 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611623 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc47d8 )
:3:hip_device_runtime.cpp   :631 : 4815821611627 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611633 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc44b4 )
:3:hip_device_runtime.cpp   :631 : 4815821611638 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821611644 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821611650 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :638 : 4815821611656 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDeviceCount ( 0x70536bbc4804 )
:3:hip_device_runtime.cpp   :640 : 4815821611660 us: [pid:2429112 tid:0x70536bbc5640] hipGetDeviceCount: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821611666 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821611669 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611674 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc47d8 )
:3:hip_device_runtime.cpp   :631 : 4815821611678 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611683 us: [pid:2429112 tid:0x70536bbc5640]  hipGetDevice ( 0x70536bbc44b4 )
:3:hip_device_runtime.cpp   :631 : 4815821611688 us: [pid:2429112 tid:0x70536bbc5640] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :653 : 4815821611694 us: [pid:2429112 tid:0x70536bbc5640]  hipSetDevice ( 0 )
:3:hip_device_runtime.cpp   :657 : 4815821611698 us: [pid:2429112 tid:0x70536bbc5640] hipSetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611742 us: [pid:2429112 tid:0x7056c923d440]  hipGetDevice ( 0x7fffacce439c )
:3:hip_device_runtime.cpp   :631 : 4815821611750 us: [pid:2429112 tid:0x7056c923d440] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :623 : 4815821611758 us: [pid:2429112 tid:0x7056c923d440]  hipGetDevice ( 0x7fffacce44fc )
:3:hip_device_runtime.cpp   :631 : 4815821611762 us: [pid:2429112 tid:0x7056c923d440] hipGetDevice: Returned hipSuccess :
:3:hip_device_runtime.cpp   :608 : 4815821611768 us: [pid:2429112 tid:0x7056c923d440]  hipDeviceSynchronize (  )
:3:rocvirtual.hpp           :66  : 4815821611780 us: [pid:2429112 tid:0x7056c923d440] Host active wait for Signal = (0x7054affff300) for -1 ns
Memory access fault by GPU node-2 (Agent handle: 0x6fec970) on address 0x705065c01000. Reason: Unknown.
Aborted (core dumped)

authors: antiagainst garrettbyrd

repo: triton-lang/triton


text:
Parabolic mortars for P4estMesh

This PR should not be merged until https://github.com/trixi-framework/Trixi.jl/pull/1629 is merged.
This overlaps with the PR in https://github.com/trixi-framework/Trixi.jl/pull/1661 as well

Thanks a lot for tackling this @jlchan @apey236 ! I left some suggestions where I spotted some debugging left-overs from my attempts.

@jlchan How bad are the merge conflicts? Does it make sense to stay with this or start from main again?

A reduced version of this PR is https://github.com/trixi-framework/Trixi.jl/pull/1691 which is forked from this to maintain commits from @jlchan and @apey236 .

@jlchan How bad are the merge conflicts? Does it make sense to stay with this or start from main again?

I don't think they're bad. Since your PR supercedes ours, we should mostly just be using your commits.
Thanks for making the second PR. I'll take a look hopefully tomorrow.

With https://github.com/trixi-framework/Trixi.jl/pull/1691 being merged, this should be reduced to only adding the 3D part.

@apey236 would you mind making a separate PR for your 3D parabolic mortar implementation? We can then close this PR

@apey236 would you mind making a separate PR for your 3D parabolic mortar implementation? We can then close this PR

@jlchan Sure I will.

source: gharchive/pull-request

authors: DanielDoehring apey236 jlchan

license: MIT

license_type: permissive

license_source: github-api


text:
Keep VERSION consistent to LibTrixi.jl/Project.toml

Addendum to #56 : Do we need and can we make sure that the versions in file VERSION  and LibTrixi.jl/Project.toml agree?

I've been thinking about this a little more. One could consider that LibTrixi.jl is really the main software product here, and that everything else is just a means to use it, ie, by providing C and Fortran bindings. In that case, we could remove the VERSION file and have the existing trixi_version_xxx functions call corresponding functions in LibTrixi.jl (as opposed to having hard coded return values as they do now).
This would also resolve #57, since we could just rely on JuliaRegistrator and TagBot to do the release process for us.

Sounds reasonable, fine with me.

source: gharchive/issue

authors: bgeihe sloede

repo: trixi-framework/libtrixi


text:
CGI scripts, specifically cgit

I am trying to get cgit to be served by merecat and have come up against this bug. I do not fully understand CGI and PATH_INFO but I did track down an old patch for thttpd which seems relevant. However, I cannot figure out if this patch has been applied as it cannot be applied in its current form.
I'd be happy to provide any other information to help debug this.

Hi! What version of Mercat httpd are you using, a released version or the latest master? I've mostly run GitWeb and fixed some CGI issues surrounding that, see http://git.troglobit.com/
The mail thread you linked to look very interesting, I found the "titan" patches¹ which I'll look into when I find the time. There could definitely be more details to look into. The patch itself is however already applied to Merecat, which leads me back to my first question; what version of Merecat are you using?

¹) https://marc.info/?l=thttpd&m=108352795820187&w=2

What version of Mercat httpd are you using, a released version or the latest master?

Latest master, built into a debian package. Its running on armhf.

OK, thanks!  I think I've reproduced your problem now.  Set up a cgit instance on my own server http://cgit.troglobit.com/ ... I'll look into the details over the weekend.

With the following in my /etc/cgitrc, a bit of linking+symlinking, and some magic virtual host bind-mounts, I got it working on my Ubuntu server
# cgit config

css=/cgit.css
logo=/cgit.png

# Prepend this to all URLs
virtual-root=cgit.cgi/

# Home of all GIT repos
scan-path=/srv/git/

# if you do not want that webcrawler (like google) index your site
robots=noindex, nofollow

My /usr/share/cgit is bind mounted into /var/www/cgit.troglobit.com and contains the following files (cgit.cgi is linked from ../../lib/cgit/):
root@example:/usr/share/cgit# ll
total 1000
drwxr-xr-x   2 root root   4096 Mar 30 11:16 ./
drwxr-xr-x 140 root root   4096 Mar 30 09:39 ../
-rwxr-xr-x   2 root root 986800 Feb 12  2016 cgit.cgi*
-rw-r--r--   1 root root  13312 Feb 12  2016 cgit.css
-rw-r--r--   1 root root   1278 Feb 12  2016 cgit.png
-rw-r--r--   1 root root   1078 Feb 12  2016 favicon.ico
lrwxrwxrwx   1 root root     22 Mar 30 11:16 filters -> ../../lib/cgit/filters/
lrwxrwxrwx   1 root root      8 Mar 30 10:16 index.cgi -> cgit.cgi*
-rw-r--r--   1 root root     47 Feb 12  2016 robots.txt

Thanks @troglobit. I tried your config but I could not get it working without vhosts. I cannot setup vhosts at this time but I will sooner or later.

Aha, well that could be that you need to use /var/www/cgi-bin/ for all such purposes, or an issue with chrooting.

authors: alephnull troglobit

repo: troglobit/merecat