text:
Pico FatFS_SPI_example.elf locks on strcmp()

This is odd, and it took a while to sort out, but your example (FatFS_SPI_example) locks when the strcmp() function is reached (in sd_get_by_name(), sd_get_fs_by_name() and process_stdio()).
I don't know if this is the result of using 2 picos (one as a debug probe) that somehow interferes (that makes no sense, but all strings appear nul-terminated, either explicitly or by calling memset())
However, there is no doubt what is occurring. I haven't taken the time to chase down why strcmp() was hardlocking on comparisons, I simply replaced the three calls to strcmp() with calls to strcmp2() and supplied my own function, e.g.
/** string compare without string.h */
static int strcmp2 (const char *a, const char *b)
{
    if (!a && !b) return 0;
    if ( a && !b) return 1;
    if (!a &&  b) return -1;

    for (; *a && *a == *b; a++, b++) {}

    return (*a > *b) - (*a < *b);
}

Example works flawlessly thereafter:
> setrtc 24 11 23 6 55 0

> date
Fri Nov 24 06:55:03 2023
Day of year: 328

> mount 0:

> ls
Directory Listing: 0:/
System Volume Information [directory] [size=0]
filename.txt [writable file] [size=56]
numbers.txt [writable file] [size=61]

> cat numbers.txt
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10

> getfree
  31248384 KiB total drive space.
  31248192 KiB available.

>

If I get more time I'll go through where your strcmp() breaks down, and if I find the issue, I'll create a pull-request. I did go ahead and ensure the inputs in cmd[] were all nul-terminated without relying on memset() (though that should also work). The change made there just ensures the nul-character is written after each character added to cmd[], e.g.:
...
    else {  // Not newline
        if (cRxedChar == '\b' || cRxedChar == (char)127) {
            /* Backspace was pressed.  Erase the last character
             in the string - if any. */
            if (ix > 0) {
                ix--;
                // cmd[ix] = '\0';
            }
        }
        else {
            /* A character was entered.  Add it to the string
             entered so far.  When a \n is entered the complete
             string will be passed to the command interpreter. */
            if (ix < sizeof cmd - 1) {
                cmd[ix] = cRxedChar;
                ix++;
            }
        }
        cmd[ix] = '\0';
    }
...

Otherwise, great job on the example! It was quite helpful.
One other little nit is there is an encoding problem with the degree symbol in data_log_demo.c, that just results in some interesting output, e.g.
> cat /data/2023-11-24/07.csv
Date,Time,Temperature (��C)
2023-11-24,07:06:32,67.4
2023-11-24,07:06:33,74.9
2023-11-24,07:06:34,74.9
2023-11-24,07:06:35,74.9

Thanks again.

This is odd, and it took a while to sort out, but your example (FatFS_SPI_example) locks when the strcmp() function is reached (in sd_get_by_name(), sd_get_fs_by_name() and process_stdio()).

I can't explain that.

I don't know if this is the result of using 2 picos (one as a debug probe) that somehow interferes (that makes no sense, but all strings appear nul-terminated, either explicitly or by calling memset())

I, too, use two Picos (one as a picoprobe).

Example works flawlessly thereafter:

This is what I see at commit 8d44d16:

> setrtc 24 11 23 16 6 0

> date
Fri Nov 24 16:06:02 2023
Day of year: 328

> mount 0:

> ls
Directory Listing: 0:/
System Volume Information [directory] [size=0]
bench.dat [writable file] [size=5242880]
bf2 [writable file] [size=10485760]
numbers.txt [writable file] [size=62]
bf [writable file] [size=41943040]
 ë [writable file] [size=736690176]
bf3 [writable file] [size=1048576]
x [writable file] [size=8388608]
y [writable file] [size=1048576]
z [writable file] [size=1048576]
a [writable file] [size=20971520]
bf80 [writable file] [size=83886080]
bf40 [writable file] [size=41943040]
bf4G [writable file] [size=4293918720]
bf20 [writable file] [size=20971520]
filename.txt [writable file] [size=14]
data [directory] [size=0]

> cat numbers.txt
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11

> getfree 0:
  15549952 KiB (15185 MiB) total drive space.
  10405568 KiB (10161 MiB) available.

One other little nit is there is an encoding problem with the degree symbol in data_log_demo.c, that just results in some interesting output...

That might be your terminal. I'm using PuTTY with "Remote character set" set to "UTF-8":
> cat data/2023-11-24/16.csv
Date,Time,Temperature (°C)
2023-11-24,16:01:25,81.4
2023-11-24,16:01:26,79.6
2023-11-24,16:01:27,80.0
2023-11-24,16:01:28,80.0
2023-11-24,16:01:29,79.6
2023-11-24,16:01:30,79.6
2023-11-24,16:01:31,79.6
2023-11-24,16:01:32,79.6
2023-11-24,16:01:33,80.0
2023-11-24,16:01:34,80.5

Yep, the encoding is the terminal. I was just using an x-term with Minimcom on Linux. I'll keep looking at the strcmp() issue.
Your example is excellent. I've tried two different breakouts (cheapo HiLetgo $1 version no card-detect, and Adafruit ADA254), both work seamlessly with SPI at 25MHz/s. I've used Samsung and PHY 32G MicroSD cards. I've used both the Pico and Pico-W, no issues. (wouldn't really expect any there)
The PHY card took writing to before it would return a getfree size. Maybe normal, or maybe just the questionable default FAT32 format on the card. I manually formatted the Samsung card, but decided to skip that step to test the PHY. Works! Only difference being the getfree on a blank (empty) card.
Keep up the great work.

I did add a 100 ms timeout in you Super Loop to tame the iterations down.

The SDK provides is stdio_set_chars_available_callback if you have something productive to do instead of iterating there. I use it in stdio_cli.c.

The PHY card took writing to before it would return a getfree size.

That's interesting. I don't have any PHY cards, let alone virgin PHY cards, to try that with. The command is just a passthrough to f_getfree in the FatFs - Generic FAT Filesystem Module, so maybe there's a bug in that library.

I've tried two different breakouts (cheapo HiLetgo $1 version no card-detect, and Adafruit ADA254), both work seamlessly with SPI at 25MHz/s.

Using my Pico Stackable, Plug & Play SD Card Expansion Module I normally run at 31250000 Hz (default 125 MHz system clock speed / 4), configured as in exp_mod_A_SPI.hw_config.c.

The character encoding of the °C symbol is a result of the pico-sdk stdio.h output. Changing to KDE konsole, the output is:
> cat data/2023-11-24/15.csv
Date,Time,Temperature (�°C)

But the terminal is fine with the UTF-8 hex codepoint of 0xc2b0 (which is in fact what is stored in the data file). Writing from glibc has no issue, e.g.
#include <stdio.h>

int main (void) {
  puts ("The UTF-8 degree '°' in degrees C is simply °C");
  return 0;
}

Output:
$ ./bin/degCutf-8
The UTF-8 degree '°' in degrees C is simply °C

So the issue is pico-sdk related in how it outputs UTF-8 on Linux.
Aside - quick fix to prevent the cursor from backspacing over prompt when deleting characters, in Super Loop:
...
    else {  /* Not newline, check backspace and delete */
        if (cRxedChar == '\b' || cRxedChar == 127) {
            /* Backspace was pressed.  Erase the last character
             in the string - if any. */
            if (ix > 0) {
                ix--;
                /* added to overwrite character removed */
                if (cRxedChar == 127) {     /* if del key, backup */
                    putchar ('\b');
                }
                putchar (' ');              /* overwrite displayed char */
                putchar ('\b');             /* backup again */
            }
            else if (cRxedChar == '\b') {   /* prevent backspace over prompt */
                putchar (' ');
            }
        }
        else {
            /* A character was entered.  Add it to the string
             entered so far.  When a \n is entered the complete
             string will be passed to the command interpreter. */
            if (ix < sizeof cmd - 1) {
                cmd[ix] = cRxedChar;
                ix++;
            }
        }
        cmd[ix] = '\0';       /* explicitly nul-terminate */
    }
...

Now pressing backspace as many times as you like won't overwrite the cursor on the line. (I apologize for the tangent, I tinker as I'm working though code... but worth passing along. You can delete or tinker yourself if you like)

This is odd, and it took a while to sort out, but your example (FatFS_SPI_example) locks when the strcmp() function is reached (in sd_get_by_name(), sd_get_fs_by_name() and process_stdio()).

It might be that it is running out of stack space. You could try adding something like:
# https://datasheets.raspberrypi.com/pico/raspberry-pi-pico-c-sdk.pdf
target_compile_definitions(FatFS_SPI_example PRIVATE
  PICO_STACK_SIZE=0x1000
)

to CMakeLists.txt.

That is a good thought, but nope. The CMakeLists.txt for the new no-OS-FatFS-SD-SDIO-SPI-RPi-Pico command_line example contains:
# https://datasheets.raspberrypi.com/pico/raspberry-pi-pico-c-sdk.pdf
target_compile_definitions(command_line PRIVATE
  PICO_STACK_SIZE=0x1000
)

And that example locks up at the exact same point. I'll keep digging -- I got side-tracked when I found your ...SDIO... repo and had to pull and check that out. Only bear was you changed the gpio pins -- presumably so you could have sequential gpios for the SDIO mode (which I haven't tried yet, just built the simple example and command_line example and now I'll go back and fix the strcmp() issue in the command_line example.
Not sure why you chose C++ for FatFsSD.cpp code (yes, std::vector is nice, but the compile time and memory overhead is a tradeoff. I'll report back when I get further into the strcmp() issue.

Will add all and report back. Have had quite a bit of fun with both versions of the repo, the original and the SDIO. The bench example also hands (but does not lock). The 5M bench.dat is properly created, not errors are received, it just doesn't output anything after:
> bench
Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0x1b
OEM ID: SM
Product: JB1RT
Revision: 3.0
Serial number: 0x6bed5f60
Manufacturing date: 6/2021

FILE_SIZE_MB = 5
BUF_SIZE = 20480
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec

Simple pressing [Enter] provides a new fully functional prompt. I added a bit of debug output in the nTest loops AND simply providing the output provided enough delay to solve the issue. Had baud at 25 * 1000 * 1000. Removed debug output and dropped baud to 12 * 1000 * 1000 -- no change, same stall.
I added the debug output in the nTest loop, before and after the n loop, e.g.
...
        t = millis();
#ifdef DCRDEBUG
        /* DEBUG */
        IMSG_PRINTF("rewind complete, beginning nTest: %hhu\n", nTest);
#endif
        for (uint32_t i = 0; i < n; i++) {
            uint32_t m = micros();
            unsigned int bw;
            fr = f_write(&file, buf, BUF_SIZE, &bw); /* Write it to the destination file */
...
            }
#ifdef DCRDEBUG
            /* DEBUG */
            if (i && i % 10 == 0) {
              IMSG_PRINTF("i: %lu  %lu,%lu\n", i, maxLatency, minLatency);
            }
#endif
        }

Surprisingly, all works fine with the slight delay caused by the debug output, e.g.
> bench
Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0x1b
OEM ID: SM
Product: JB1RT
Revision: 3.0
Serial number: 0x6bed5f60
Manufacturing date: 6/2021

FILE_SIZE_MB = 5
BUF_SIZE = 20480
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
rewind complete, beginning nTest: 0
i: 10  15265,11024
i: 20  15265,11024
i: 30  15265,11017
i: 40  15265,11017
i: 50  15265,11017
...
1779.1,15265,10895,11478
rewind complete, beginning nTest: 1
i: 10  12590,11018
i: 20  12590,11015
i: 30  12590,11015
i: 40  12590,11015
i: 50  12590,11015
...
1782.1,14643,10943,11463

Starting read test, please wait.

So there is something strange happening that seems timing related in that loop.
Humor... I can't tell you how much time you can waste digging through cmake files trying to sort out a build failure due to a typo of FatFS_SPI in a CMakeLists.txt instead of FatFs_SPI.... That's one I can look at all day and never see.... Good to be human.

Maybe your terminal is sending extraneous NUL characters. Does it help if you add a check like this:
    if (!(0 < cRxedChar &&  cRxedChar < 0x7F))
        return; // Not dealing with multibyte characters

to the beginning of void process_stdio(int cRxedChar)?
And maybe
    assert(cmd);
    assert(cmd[0]);

to the beginning of static void process_cmd(char *cmd)?

Added both, e.g.
static void process_cmd(char *cmd) {
    assert(cmd);
    assert(cmd[0]);

    char *cmdn = strtok_r(cmd, " ", &saveptr);
    if (cmdn) {
...

and
void process_stdio(int cRxedChar) {
    static char cmd[256];
    static size_t ix;

    if (!(0 < cRxedChar &&  cRxedChar < 0x7F)) {
        printf ("error: hist multibyte in %s()\n", __func__);
        return; // Not dealing with multibyte characters
    }
...

Same hardlock as soon as strcmp() is called. I'm going to add debug output to output the cmd each time and see if I can identify any other possible explanation. In this latest test, simply running any command causes the lock when strcmp() is called, e.g.
> setrtc 4 12 23 4 46 15

(hardlocked at that point) :(

God windows is a PITA. Had openocd built, but didn't have zadig installed yet, then Msys64 has some odd mount of the filesystem so if you have the cloned repo in one directory, but you copy/build the command_line example from another directory under your home, then cmake refuses to find the CMakeLists.txt under src/, etc...
But, there are no issues with strcmp() under windows. and no stall of bench. However, a backspace or other non-character input does trigger a multi-byte character warning. It was good to see it all just work :)
> setrtc 4 12 23 21 56 15

> date
Mon Dec  4 21:56:20 2023
Day of year: 338

> mount 0:

> ls
Directory Listing: 0:/
System Volume Information [directory] [size=0]
bench.dat [writable file] [size=5242880]
numbers.txt [writable file] [size=61]
data [directory] [size=0]
cdef [directory] [size=0]
simple-SDIO.txt [writable file] [size=14]
simple.txt [writable file] [size=70]

> bench 0:
Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0x1b
OEM ID: SM
Product: JB1RT
Revision: 3.0
Serial number: 0x6bed5f60
Manufacturing date: 6/2021

FILE_SIZE_MB = 5
BUF_SIZE = 20480
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1792.4,17819,10846,11411
1790.6,14693,10925,11425

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1927.5,11495,10263,10622
1928.2,11496,10264,10620

Done

>

This points to either an encoding issue or a line-ending issue. This may be tricky to debug if the issue is triggered in glibc. I can identify where it is failing, but now why yet. I'll try dumping the characters in cmd and see if that discloses any more information.

This is beginning to look more and more like a communications problem and the difference between how PuTTY and minicom do serial communications. It could be something as mundane as one adding a '\r' or '\n' and the other not.
I can build the example on Linux and flash and then plug the pico into a windows box and fire up Device Manager and PuTTY and the example works fine with the substitute version of strcmp(). The bench.c example runs without hanging.
I can then build on windows in Msys2 using string.h strcmp() flash and run on windows and all works fine.
Amazingly, I can take plug the pico into my Linux box and the menu works showing strcmp() is happy, but now the bench.c example hangs as described above.
Both PuTTY and minicom are using 115200 N81. PuTTY is using XON/XOFF flow control and minimum has hardware flow control DISABLED (defaulting to XON/XOFF). I try toggling Add a carriage return with combinations Add a newline, but that doesn't change anything as far as bench hanging.
Strange issue. I recall a couple of point where the code is checking for '\n', I'll take a closer look there. Head-scratcher for sure.

Can you run it under the debugger? If so, if you recreate a hang condition can you then pause it with the debugger and see the Call Stack?

All I have is openocd and gdb to work with through a 2nd terminal. I've had mixed success there stepping though larger programs (this qualifies, there are quite a few rabbit-holes to jump down). It's not cmdn or argp related in process_cmd(). Those appear fine.
I think your suspicion of this being stack related is probably right. Though why the difference in behavior between PuTTY and minicom is a head scratcher -- and why a standard function like strcmp() would hang. This is definitely a challenge.
I'll see if I can make gdb behave in openocd and step through the code until I can get it to hang.  (the loops in bench.c may be a bit tedious :)

There is very little stack protection or diagnostics with the RP2040. I'm going to look at a custom linker script to move the stack to the bottom of the memory layout. That way, if the (fixed size) stack went past the "bottom" of the RAM it would trigger a bus fault exception. As it is with the Pico SDK default linker script the stack and heap can collide with no warning.

We are on the right track. I have modified the simple examples including string.h and using libc strcmp() without any problems. So (guessing) the inclusion of the string.h code in the larger examples is causing the program to exceed either a stack or similar segment limit. Replacing the strcmp() function with my little manual check evidently brings the code side down enough to avoid the issue.
I'm using the 10.3 version of the gcc-arm-none-eabi compiler. I don't know if that matters. With Msys2 on windows it is the 13.1 versions.
I'll keep plugging away here, but I vastly under estimated the number of rabbit trials that branch from the command.c example throughout the FatFS tree. It has been a good introduction to the code, but slow going :)

I moved the core 0 stack to the bottom of RAM, and I did find some stack overflows. If you'd like to give it a try, you can checkout the dev branch of no-OS-FatFS-SD-SDIO-SPI-RPi-Pico.

Okay, I'll do that and report back. One issue I found a work around for was the °C output being wonky. That is a minicom issue. I switched to using picocom and then output is fine, e.g.
> cat data/2023-12-12/18.csv
Date,Time,Temperature (°C)
2023-12-12,18:03:16,14.0
2023-12-12,18:03:17,16.8
...

Why minicom does that is strange, but that's not an issue here. Give me a bit and I'll report back tonight.

Well, seems there is a gremlin in your new build files, everything builds fine, but then falls-over during the link, e.g.
...
[ 98%] Building C object CMakeFiles/command_line.dir/opt/pico-sdk/src/rp2_common/hardware_adc/adc.c.obj
[100%] Linking CXX executable command_line.elf
/usr/lib64/gcc/arm-none-eabi/10/../../../../arm-none-eabi/bin/ld: CMakeFiles/command_line.dir/src/command.cpp.obj: in function `run_ls(unsigned int, char const**)':
command.cpp:(.text._ZL6run_lsjPPKc+0xc): undefined reference to `ls'
/usr/lib64/gcc/arm-none-eabi/10/../../../../arm-none-eabi/bin/ld: command.cpp:(.text._ZL6run_lsjPPKc+0x14): undefined reference to `ls'
/usr/lib64/gcc/arm-none-eabi/10/../../../../arm-none-eabi/bin/ld: CMakeFiles/command_line.dir/tests/simple.c.obj: in function `simple':
simple.c:(.text.simple+0x128): undefined reference to `ls'
/usr/lib64/gcc/arm-none-eabi/10/../../../../arm-none-eabi/bin/ld: simple.c:(.text.simple+0x13a): undefined reference to `ls'
Memory region         Used Size  Region Size  %age Used
           FLASH:      201976 B         2 MB      9.63%
       SCRATCH_X:          0 GB         4 KB      0.00%
       SCRATCH_Y:          0 GB         4 KB      0.00%
             RAM:       29588 B       248 KB     11.65%
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/command_line.dir/build.make:1704: command_line.elf] Error 1
make[1]: *** [CMakeFiles/Makefile2:1554: CMakeFiles/command_line.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

I'll see if I can figure out why the linker is unhappy with ls and run_ls...

Program runs and displays the prompt, but fails for anything typed at the command line with:
*** Fault Capture Analysis (RAM): ***
        Event:  CM4 Hard Fault.
        Time: 1970-01-01 00:00:00
                Link Register (LR): 0x1000154F
                Program Counter (PC): 0x00000000

That happens for anything. date, setrtc, "foo", anything -- this I'm not sure where to start to fix :(

Okay, I'll do that and report back. One issue I found a work around for was the °C output being wonky. That is a minicom issue. I switched to using picocom and then output is fine, e.g.
...
Why minicom does that is strange, but that's not an issue here. Give me a bit and I'll report back tonight.

You might need to configure minicom to support UTF-8. See https://www.wilivm.com/blog/enabling-utf-8-in-minicom-on-linux/

Well, seems there is a gremlin in your new build files, everything builds fine, but then falls-over during the link, e.g.

...

Uhhhh Didn't Find It (literally -- it's missing from command.cpp) and the rest of the files, e.g.

...

In the old command.cpp it was defined. (I'll just copy from there for purposes of this test)

Oops. I missed checking in the file f_util.c. I decided to move the ls function there since it might be useful in other contexts besides the command_line example (e.g., it is used in the simple test).

Program runs and displays the prompt, but fails for anything typed at the command line with:
*** Fault Capture Analysis (RAM): ***
        Event:  CM4 Hard Fault.
        Time: 1970-01-01 00:00:00
                Link Register (LR): 0x1000154F
                Program Counter (PC): 0x00000000

Looking at build\command_line.dis that "Link Register (LR): 0x1000154F" address seems to be in dump_crash_info , which is one of the first things called in main. Seems like it crashed trying to analyze a preceding crash.

That happens for anything. date, setrtc, "foo", anything -- this I'm not sure where to start to fix :(

Sounds like anything much going on the stack is causing a crash. Which, in a way, is progress, since we previously suspected some kind of stack corruption problem.
If you look at build\command_line.elf.map where is your __StackBottom and __StackTop? Mine looks like this:
.stack1_dummy   0x20001000        0x0
                0x20001000                __StackOneBottom = .
 *(.stack1*)
                0x20001000                __StackOneTop = .

.stack_dummy    0x20000000     0x1000
                0x20000000                __StackBottom = .
 *(.stack*)
 .stack         0x20000000     0x1000 CMakeFiles/command_line.dir/C_/Program_Files/Raspberry_Pi/Pico_SDK_v1.5.1/pico-sdk/src/rp2_common/pico_standard_link/crt0.S.obj
                0x20001000                __StackTop = .
                0x20001000                PROVIDE (__stack = __StackTop)

which is a 4 kB stack (PICO_STACK_SIZE=0x1000) at the bottom of RAM, 0x20000000 (see linker\memmap.ld).

Huh? 64-bit addresses? Maybe just a different in the platform generated output on Linux, but I have:
stack1_dummy   0x0000000020001000        0x0
                0x0000000020001000                __StackOneBottom = .
 *(.stack1*)
                0x0000000020001000                __StackOneTop = .

.stack_dummy    0x0000000020000000     0x1000
                0x0000000020000000                __StackBottom = .
 *(.stack*)
 .stack         0x0000000020000000     0x1000 CMakeFiles/command_line.dir/opt/pico-sdk/src/rp2_common/pico_standard_link/crt0.S.obj
                0x0000000020001000                __StackTop = .
                0x0000000020001000                PROVIDE (__stack = __StackTop)

Which is right inline with what you have only a difference in pointer-size. I've never looked at a .map file before, but the address size is the same in all my .map files, so that's not a problem per-se.

Fresh pull of the dev branch and rebuild of the command_line example - same result in the terminal window:
*** Fault Capture Analysis (RAM): ***
        Event:  CM4 Hard Fault.
        Time: 1970-01-01 00:00:00
                Link Register (LR): 0x10001447
                Program Counter (PC): 0x00000000

>

Here is everything from the .map file from *fill* to flash_end:
 *fill*         0x0000000020003e86        0x2
                0x0000000020003e88                __bss_end__ = .

.heap           0x0000000020003e88      0x800
                0x0000000020003e88                __end__ = .
                0x0000000020003e88                end = __end__
 *(.heap*)
 .heap          0x0000000020003e88      0x800 CMakeFiles/command_line.dir/opt/pico-sdk/src/rp2_common/pico_standard_link/crt0.S.obj
                0x0000000020004688                __HeapLimit = .

.stack1_dummy   0x0000000020001000        0x0
                0x0000000020001000                __StackOneBottom = .
 *(.stack1*)
                0x0000000020001000                __StackOneTop = .

.stack_dummy    0x0000000020000000     0x1000
                0x0000000020000000                __StackBottom = .
 *(.stack*)
 .stack         0x0000000020000000     0x1000 CMakeFiles/command_line.dir/opt/pico-sdk/src/rp2_common/pico_standard_link/crt0.S.obj
                0x0000000020001000                __StackTop = .
                0x0000000020001000                PROVIDE (__stack = __StackTop)

.flash_end      0x000000001002a3fc        0x0
                0x000000001002a3fc                PROVIDE (__flash_binary_end = .)
                0x0000000020040000                __StackLimit = (ORIGIN (RAM) + LENGTH (RAM))
                0x0000000000000001                ASSERT (((__binary_info_header_end - __logical_binary_start) <= 0x100), Binary info must be in first 256 bytes of the binary)

I have a hunch that you might be getting spurious interrupts on the card detect GPIO. I was trying to think of what could be eating up stack space, and it occurred to me that the card_detect_callback was doing way more than an interrupt service routine should. I pushed a change to the "dev" branch that defers the processing to the superloop.

Give me a sec and I'll report back.

You could try running with openocd and gdb and get an idea what's on the stack. I did
    (gdb) target remote localhost:3333
    (gdb) load
    (gdb) monitor reset init

Then
(gdb) info registers

shows the Stack Pointer
...
sp             0x20000eb0          0x20000eb0
...

at  0x20000eb0, so there are 0x20001000 - 0x20000eb0 = 0x150, or 336 bytes in use at the entry point. Then I did
(gdb) b run_help
(gdb) cont
Continuing.

On the terminal, I enter help, which takes me to
Thread 1 "rp2040.core0" hit Breakpoint 1, run_help (argc=0, argv=0x20000e9c)
    at C:/Users/carlk/pi/pico/work/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/src/command.cpp:736
736     static void run_help(const size_t argc, const char *argv[]) {
(gdb) where
#0  run_help (argc=0, argv=0x20000e9c)
    at C:/Users/carlk/pi/pico/work/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/src/command.cpp:736
#1  0x10000764 in process_cmd (cmd_sz=cmd_sz@entry=256, cmd=cmd@entry=0x200082e4 <process_stdio(int)::cmd> "help")
    at C:/Users/carlk/pi/pico/work/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/src/command.cpp:792
#2  0x100012d2 in process_stdio (cRxedChar=<optimized out>)
    at C:/Users/carlk/pi/pico/work/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/src/command.cpp:825
#3  0x1000048c in main ()
    at C:/Users/carlk/pi/pico/work/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/main.cpp:98

and now
(gdb) info registers

shows sp at 0x20000e98, so there are 0x20001000 - 0x20000e98 = 0x168 (360) bytes in use.

Sure, give me a minute to get me openocd cheatsheet up and I'll give it a go. Was about to go pull everything on windows and confirm build there. We will put that on hold and try the debug.

Here is a second run through:
(gdb) load
Loading section .boot2, size 0x100 lma 0x10000000
Loading section .text, size 0x22c78 lma 0x10000100
Loading section .rodata, size 0x6090 lma 0x10022d78
Loading section .binary_info, size 0x28 lma 0x10028e08
Loading section .data, size 0x15dc lma 0x10028e30
Start address 0x100001e8, load size 173068
Transfer rate: 28 KB/sec, 12362 bytes/write.
(gdb) monitor reset init
target halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ea msp: 0x20041f00
(gdb) info registers
r0             0x0                 0
r1             0x10                16
r2             0x40034000          1073954816
r3             0x197               407
r4             0x20000e9f          536874655
r5             0x1                 1
r6             0x5f3de8d           99868301
r7             0x0                 0
r8             0x20003e0c          536886796
r9             0x0                 0
r10            0x2c0               704
r11            0x20003878          536885368
r12            0x0                 0
sp             0x20000e94          0x20000e94
lr             0x1001ad9f          268545439
pc             0x100001e8          0x100001e8 <_entry_point>
xPSR           0x21000000          553648128
msp            0x20000e94          0x20000e94
psp            0xfffffffc          0xfffffffc
primask        0x0                 0
basepri        0x0                 0
faultmask      0x0                 0
control        0x0                 0
(gdb) b run_help
Note: breakpoint 1 also set at pc 0x10000ac6.
Breakpoint 2 at 0x10000ac6
(gdb) cont
Continuing.
rp2040.core0: external reset detected
^C
Program received signal SIGINT, Interrupt.
0x1001ad66 in getchar_timeout_us ()
(gdb) backtrace
#0  0x1001ad66 in getchar_timeout_us ()
#1  0x10000420 in main ()
(gdb) frame 0
#0  0x1001ad66 in getchar_timeout_us ()
(gdb) info frame
Stack level 0, frame at 0x20000ec8:
 pc = 0x1001ad66 in getchar_timeout_us; saved pc = 0x10000420
 called by frame at 0x20001000
 Arglist at 0x20000ea0, args:
 Locals at 0x20000ea0, Previous frame's sp is 0x20000ec8
 Saved registers:
  r4 at 0x20000eb4, r5 at 0x20000eb8, r6 at 0x20000ebc, r7 at 0x20000ec0, r8 at 0x20000eac, r9 at 0x20000eb0,
  lr at 0x20000ec4
(gdb) frame 1
#1  0x10000420 in main ()
(gdb) info frame
Stack level 1, frame at 0x20001000:
 pc = 0x10000420 in main; saved pc = 0x10000222
 caller of frame at 0x20000ec8
 Arglist at 0x20000ec8, args:
 Locals at 0x20000ec8, Previous frame's sp is 0x20001000
 Saved registers:
  r4 at 0x20000fec, r5 at 0x20000ff0, r6 at 0x20000ff4, r7 at 0x20000ff8, r8 at 0x20000fdc, r9 at 0x20000fe0,
  r10 at 0x20000fe4, r11 at 0x20000fe8, lr at 0x20000ffc

Looks much the same. I'll try stepping and see where I get to.

Maybe instead of doing b run_help, instead b process_cmd,  cont to there, then start stepping. If you can find which step faults, get back to just before that step and display/i $pc then si (and enter to repeat).

Hmm, rather around about debug, but the surprise was  process_cmd asking if it was in a future library? I went through the session, set the program counter to display and then step instructions until it begins looping waiting for input, e.g.
(gdb) monitor reset init
target halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ea msp: 0x20041f00
(gdb) b process_cmd
Function "process_cmd" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (process_cmd) pending.
(gdb) info registers
r0             0xfffffffd          -3
r1             0x10                16
r2             0x0                 0
r3             0xa6050a            10880266
r4             0x0                 0
r5             0x20003e10          536886800
r6             0xa60509            10880265
r7             0x0                 0
r8             0x20003e0c          536886796
r9             0x0                 0
r10            0x2c0               704
r11            0x20003878          536885368
r12            0x0                 0
sp             0x20000ea0          0x20000ea0
lr             0x1001ad9f          268545439
pc             0x1001add0          0x1001add0 <getchar_timeout_us+132>
xPSR           0x21000000          553648128
msp            0x20041f00          0x20041f00
psp            0xfffffffc          0xfffffffc
primask        0x0                 0
basepri        0x0                 0
faultmask      0x0                 0
control        0x0                 0
(gdb) display/i $pc
1: x/i $pc
=> 0x1001add0 <getchar_timeout_us+132>: movs    r0, #1
(gdb) si
...
=> 0x24f8:      subs    r3, #1
(gdb)
0x000024fa in ?? ()
1: x/i $pc
=> 0x24fa:      bne.n   0x24f8
(gdb)
0x000024f8 in ?? ()
1: x/i $pc
=> 0x24f8:      subs    r3, #1
(gdb)
0x000024fa in ?? ()
1: x/i $pc
=> 0x24fa:      bne.n   0x24f8
(gdb)
0x000024f8 in ?? ()
1: x/i $pc
=> 0x24f8:      subs    r3, #1
(gdb)
0x000024fa in ?? ()
1: x/i $pc
=> 0x24fa:      bne.n   0x24f8
(gdb)
0x000024f8 in ?? ()
1: x/i $pc
=> 0x24f8:      subs    r3, #1
(gdb)
0x000024fa in ?? ()
1: x/i $pc
=> 0x24fa:      bne.n   0x24f8
(gdb)
0x000024f8 in ?? ()
1: x/i $pc
=> 0x24f8:      subs    r3, #1
(gdb) cont
Continuing.
rp2040.core0: external reset detected
^C
Program received signal SIGINT, Interrupt.
0x10016f78 in time_us_64 ()
1: x/i $pc
=> 0x10016f78 <time_us_64+4>:   movs    r2, r1
(gdb) backtrace
#0  0x10016f78 in time_us_64 ()
#1  0x1001ad66 in getchar_timeout_us ()
#2  0x10000420 in main ()
(gdb) frame 0
#0  0x10016f78 in time_us_64 ()
(gdb) info frame
Stack level 0, frame at 0x20000ea0:
 pc = 0x10016f78 in time_us_64; saved pc = 0x1001ad66
 called by frame at 0x20000ec8
 Arglist at 0x20000ea0, args:
 Locals at 0x20000ea0, Previous frame's sp is 0x20000ea0
(gdb) frame 1
#1  0x1001ad66 in getchar_timeout_us ()
(gdb) info frame
Stack level 1, frame at 0x20000ec8:
 pc = 0x1001ad66 in getchar_timeout_us; saved pc = 0x10000420
 called by frame at 0x20001000, caller of frame at 0x20000ea0
 Arglist at 0x20000ea0, args:
 Locals at 0x20000ea0, Previous frame's sp is 0x20000ec8
 Saved registers:
  r4 at 0x20000eb4, r5 at 0x20000eb8, r6 at 0x20000ebc, r7 at 0x20000ec0, r8 at 0x20000eac, r9 at 0x20000eb0,
  lr at 0x20000ec4
(gdb) frame 2
#2  0x10000420 in main ()
(gdb) info frame
Stack level 2, frame at 0x20001000:
 pc = 0x10000420 in main; saved pc = 0x10000222
 caller of frame at 0x20000ec8
 Arglist at 0x20000ec8, args:
 Locals at 0x20000ec8, Previous frame's sp is 0x20001000
 Saved registers:
  r4 at 0x20000fec, r5 at 0x20000ff0, r6 at 0x20000ff4, r7 at 0x20000ff8, r8 at 0x20000fdc, r9 at 0x20000fe0,
  r10 at 0x20000fe4, r11 at 0x20000fe8, lr at 0x20000ffc
(gdb) info registers
r0             0x0                 0
r1             0x0                 0
r2             0x0                 0
r3             0x40054000          1074085888
r4             0x1                 1
r5             0x10000341          268436289
r6             0x20003c70          536886384
r7             0x0                 0
r8             0x20003e81          536886913
r9             0xffffffff          -1
r10            0x2c0               704
r11            0x20003878          536885368
r12            0x0                 0
sp             0x20000ec8          0x20000ec8
lr             0x10000421          268436513
pc             0x10000420          0x10000420 <main+204>
xPSR           0x61000000          1627389952
msp            0x20000ea0          0x20000ea0
psp            0xfffffffc          0xfffffffc
primask        0x0                 0
basepri        0x0                 0
faultmask      0x0                 0
control        0x0                 0

I can see the loads, stores, moves, and branches go by, but I'm not sure what to make of the final loop -- which I took as waiting for input. That is when I continued and then typed "help" at the prompt. Same result in the terminal:
*** Fault Capture Analysis (RAM): ***
        Event:  CM4 Hard Fault.
        Time: 1970-01-01 00:00:00
                Link Register (LR): 0x10001457
                Program Counter (PC): 0x00000000

>

And then I tried to capture as much of the debug information as I could. Do you see anything in the register values or address that point to anything specific?

(gdb) b process_cmd
Function "process_cmd" not defined.

That is strange. I get:
(gdb) b process_cmd
Breakpoint 2 at 0x10000694: file C:/Users/carlk/pi/pico/work/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/src/command.cpp, line 760.

What variant of build are you using? I'm currently running a "Debug" build. If you are using the command line, that would be something like
cmake -DCMAKE_BUILD_TYPE=Debug ..

In VS Code, you can choose in the menu bar at View/Command Palette/Cmake: Select Variant.

Yes, and no, I hadn't thought to set the build variant. I just used your default CMakeLists.txt.  I have the following comments in the CMakeLists.txt template I use for the pico:
# Set build type (default is Release for Pico)
#                   GNU GCC/Clang       MSVC
#                  -----------------   ----------------------------------
#  empty            nothing set         /DWIN32 /D_WINDOWS /W3 /GR /EHsc
#  Debug            -g                  /MDd /Zi /Ob0 /Od /RTC1
#  Release          -O3 -DNDEBUG        /MD /O2 /Ob2 /DNDEBUG
#  RelWithDebInfo   -O2 -g -DNDEBUG     /MD /Zi /O2 /Ob1 /DNDEBUG
#  MinSizeRel       -Os -DNDEBUG        /MD /O1 /Ob1 /DNDEBUG
#set (CMAKE_BUILD_TYPE "Release")

Will set the debug build and report back.

What strcmp look like in build\command_line.dis? Here is mine:
1001c920 <strcmp>:
1001c920:	4602      	mov	r2, r0
1001c922:	b570      	push	{r4, r5, r6, lr}
1001c924:	430a      	orrs	r2, r1
1001c926:	0792      	lsls	r2, r2, #30
1001c928:	d12a      	bne.n	1001c980 <strcmp+0x60>
1001c92a:	4d1e      	ldr	r5, [pc, #120]	; (1001c9a4 <strcmp+0x84>)
1001c92c:	01ee      	lsls	r6, r5, #7
1001c92e:	c804      	ldmia	r0!, {r2}
1001c930:	c908      	ldmia	r1!, {r3}
1001c932:	1b54      	subs	r4, r2, r5
1001c934:	4394      	bics	r4, r2
1001c936:	4034      	ands	r4, r6
1001c938:	d017      	beq.n	1001c96a <strcmp+0x4a>
1001c93a:	b2d0      	uxtb	r0, r2
1001c93c:	b2d9      	uxtb	r1, r3
1001c93e:	1a40      	subs	r0, r0, r1
1001c940:	0621      	lsls	r1, r4, #24
1001c942:	4301      	orrs	r1, r0
1001c944:	d110      	bne.n	1001c968 <strcmp+0x48>
1001c946:	b290      	uxth	r0, r2
1001c948:	b299      	uxth	r1, r3
1001c94a:	1a40      	subs	r0, r0, r1
1001c94c:	0421      	lsls	r1, r4, #16
1001c94e:	4301      	orrs	r1, r0
1001c950:	d10a      	bne.n	1001c968 <strcmp+0x48>
1001c952:	0210      	lsls	r0, r2, #8
1001c954:	0219      	lsls	r1, r3, #8
1001c956:	0a00      	lsrs	r0, r0, #8
1001c958:	0a09      	lsrs	r1, r1, #8
1001c95a:	1a40      	subs	r0, r0, r1
1001c95c:	0221      	lsls	r1, r4, #8
1001c95e:	4301      	orrs	r1, r0
1001c960:	d102      	bne.n	1001c968 <strcmp+0x48>
1001c962:	0e10      	lsrs	r0, r2, #24
1001c964:	0e19      	lsrs	r1, r3, #24
1001c966:	1a40      	subs	r0, r0, r1
1001c968:	bd70      	pop	{r4, r5, r6, pc}
1001c96a:	429a      	cmp	r2, r3
1001c96c:	d0df      	beq.n	1001c92e <strcmp+0xe>
1001c96e:	ba10      	rev	r0, r2
1001c970:	ba19      	rev	r1, r3
1001c972:	4288      	cmp	r0, r1
1001c974:	d901      	bls.n	1001c97a <strcmp+0x5a>
1001c976:	2001      	movs	r0, #1
1001c978:	bd70      	pop	{r4, r5, r6, pc}
1001c97a:	2000      	movs	r0, #0
1001c97c:	43c0      	mvns	r0, r0
1001c97e:	bd70      	pop	{r4, r5, r6, pc}
1001c980:	7802      	ldrb	r2, [r0, #0]
1001c982:	780b      	ldrb	r3, [r1, #0]
1001c984:	3001      	adds	r0, #1
1001c986:	3101      	adds	r1, #1
1001c988:	2a00      	cmp	r2, #0
1001c98a:	d009      	beq.n	1001c9a0 <strcmp+0x80>
1001c98c:	429a      	cmp	r2, r3
1001c98e:	d107      	bne.n	1001c9a0 <strcmp+0x80>
1001c990:	7802      	ldrb	r2, [r0, #0]
1001c992:	780b      	ldrb	r3, [r1, #0]
1001c994:	3001      	adds	r0, #1
1001c996:	3101      	adds	r1, #1
1001c998:	2a00      	cmp	r2, #0
1001c99a:	d001      	beq.n	1001c9a0 <strcmp+0x80>
1001c99c:	429a      	cmp	r2, r3
1001c99e:	d0ef      	beq.n	1001c980 <strcmp+0x60>
1001c9a0:	1ad0      	subs	r0, r2, r3
1001c9a2:	bd70      	pop	{r4, r5, r6, pc}
1001c9a4:	01010101 	.word	0x01010101

I can see it gdb step through it if I set a breakpoint in strcmp, do the display/i $pc, then si.

Holy Crap!!!
Before I get to the next debug, I just wrapped everything in a #define STRCMP2 to use my little strcmp2() function, e.g.
...
#ifdef STRCMP2
            if (0 == strcmp2(cmds[i].command, cmdn)) {
#else
            if (0 == strcmp(cmds[i].command, cmdn)) {
#endif
...

and...
...

mem-stats:
 Print memory statistics

help:
 Shows this command help.

> mount 0:

> info

Manufacturer ID: 0x1b
OEM ID: SMProduct: JB1RT
Revision: 3.0
Serial number: 0x605fed6b
Manufacturing date: 6/2021

SDHC/SDXC Card: hc_c_size: 61055
Sectors: 62521344
Capacity: 30528 MiB (32010 MB)
ERASE_BLK_EN: units of 512 bytes
SECTOR_SIZE (size of an erasable sector): 128 (65536 bytes)

  31248384 KiB (30516 MiB) total drive space.
  31241696 KiB (30509 MiB) available.

Filesystem type: FAT32
Volume base sector: 8192
FAT base sector: 9316
Root directory base sector (FAT12/16) or cluster (FAT32/exFAT): 2
Data base sector: 24576
FAT Cluster size ("allocation unit"): 64 sectors (32768 bytes)

> ls
Directory Listing: 0:/
System Volume Information [directory] [size=0]
bench.dat [writable file] [size=5242880]
numbers.txt [writable file] [size=61]
data [directory] [size=0]
cdef [directory] [size=0]
simple-SDIO.txt [writable file] [size=14]
simple.txt [writable file] [size=70]
minimal.txt [writable file] [size=28]

AARRRGH!
Now I'll go debug :)

Much different .dis
1001a254 <strcmp>:
1001a254:       e020c001        eor     ip, r0, r1
1001a258:       e31c0003        tst     ip, #3
1001a25c:       1a000021        bne     1001a2e8 <strcmp+0x94>
1001a260:       e210c003        ands    ip, r0, #3
1001a264:       e3c00003        bic     r0, r0, #3
1001a268:       e3c11003        bic     r1, r1, #3
1001a26c:       e4902004        ldr     r2, [r0], #4
1001a270:       04913004        ldreq   r3, [r1], #4
1001a274:       0a000006        beq     1001a294 <strcmp+0x40>
1001a278:       e22cc003        eor     ip, ip, #3
1001a27c:       e3e034ff        mvn     r3, #-16777216  ; 0xff000000
1001a280:       e1a0c18c        lsl     ip, ip, #3
1001a284:       e1a0cc33        lsr     ip, r3, ip
1001a288:       e4913004        ldr     r3, [r1], #4
1001a28c:       e182200c        orr     r2, r2, ip
1001a290:       e183300c        orr     r3, r3, ip
1001a294:       e52d4004        push    {r4}            ; (str r4, [sp, #-4]!)
1001a298:       e3a04001        mov     r4, #1
1001a29c:       e1844404        orr     r4, r4, r4, lsl #8
1001a2a0:       e1844804        orr     r4, r4, r4, lsl #16
1001a2a4:       e042c004        sub     ip, r2, r4
1001a2a8:       e1520003        cmp     r2, r3
1001a2ac:       01ccc002        biceq   ip, ip, r2
1001a2b0:       011c0384        tsteq   ip, r4, lsl #7
1001a2b4:       04902004        ldreq   r2, [r0], #4
1001a2b8:       04913004        ldreq   r3, [r1], #4
1001a2bc:       0afffff8        beq     1001a2a4 <strcmp+0x50>
1001a2c0:       e1a00c02        lsl     r0, r2, #24
1001a2c4:       e1a02422        lsr     r2, r2, #8
1001a2c8:       e3500001        cmp     r0, #1
1001a2cc:       21500c03        cmpcs   r0, r3, lsl #24
1001a2d0:       01a03423        lsreq   r3, r3, #8
1001a2d4:       0afffff9        beq     1001a2c0 <strcmp+0x6c>
1001a2d8:       e20330ff        and     r3, r3, #255    ; 0xff
1001a2dc:       e0630c20        rsb     r0, r3, r0, lsr #24
1001a2e0:       e49d4004        pop     {r4}            ; (ldr r4, [sp], #4)
1001a2e4:       e12fff1e        bx      lr
1001a2e8:       e3100003        tst     r0, #3
1001a2ec:       0a000006        beq     1001a30c <strcmp+0xb8>
1001a2f0:       e4d02001        ldrb    r2, [r0], #1
1001a2f4:       e4d13001        ldrb    r3, [r1], #1
1001a2f8:       e3520001        cmp     r2, #1
1001a2fc:       21520003        cmpcs   r2, r3
1001a300:       0afffff8        beq     1001a2e8 <strcmp+0x94>
1001a304:       e0420003        sub     r0, r2, r3
1001a308:       e12fff1e        bx      lr
1001a30c:       e92d0030        push    {r4, r5}
1001a310:       e3a04001        mov     r4, #1
1001a314:       e1844404        orr     r4, r4, r4, lsl #8
1001a318:       e1844804        orr     r4, r4, r4, lsl #16
1001a31c:       e4902004        ldr     r2, [r0], #4
1001a320:       e2015003        and     r5, r1, #3
1001a324:       e3c11003        bic     r1, r1, #3
1001a328:       e4913004        ldr     r3, [r1], #4
1001a32c:       e3550002        cmp     r5, #2
1001a330:       0a000017        beq     1001a394 <strcmp+0x140>
1001a334:       8a00002d        bhi     1001a3f0 <strcmp+0x19c>
1001a338:       e3c254ff        bic     r5, r2, #-16777216      ; 0xff000000
1001a33c:       e1550423        cmp     r5, r3, lsr #8
1001a340:       e042c004        sub     ip, r2, r4
1001a344:       e1ccc002        bic     ip, ip, r2
1001a348:       1a000007        bne     1001a36c <strcmp+0x118>
1001a34c:       e01cc384        ands    ip, ip, r4, lsl #7
1001a350:       04913004        ldreq   r3, [r1], #4
1001a354:       1a000006        bne     1001a374 <strcmp+0x120>
1001a358:       e0255002        eor     r5, r5, r2
1001a35c:       e1550c03        cmp     r5, r3, lsl #24
1001a360:       1a000008        bne     1001a388 <strcmp+0x134>
1001a364:       e4902004        ldr     r2, [r0], #4
1001a368:       eafffff2        b       1001a338 <strcmp+0xe4>
1001a36c:       e1a03423        lsr     r3, r3, #8
1001a370:       ea000036        b       1001a450 <strcmp+0x1fc>
1001a374:       e3dcc4ff        bics    ip, ip, #-16777216      ; 0xff000000
1001a378:       1a000031        bne     1001a444 <strcmp+0x1f0>
1001a37c:       e5d13000        ldrb    r3, [r1]
1001a380:       e1a05c22        lsr     r5, r2, #24
1001a384:       ea000031        b       1001a450 <strcmp+0x1fc>
1001a388:       e1a05c22        lsr     r5, r2, #24
1001a38c:       e20330ff        and     r3, r3, #255    ; 0xff
1001a390:       ea00002e        b       1001a450 <strcmp+0x1fc>
1001a394:       e1a05802        lsl     r5, r2, #16
1001a398:       e042c004        sub     ip, r2, r4
1001a39c:       e1a05825        lsr     r5, r5, #16
1001a3a0:       e1ccc002        bic     ip, ip, r2
1001a3a4:       e1550823        cmp     r5, r3, lsr #16
1001a3a8:       1a00000e        bne     1001a3e8 <strcmp+0x194>
1001a3ac:       e01cc384        ands    ip, ip, r4, lsl #7
1001a3b0:       04913004        ldreq   r3, [r1], #4
1001a3b4:       1a000004        bne     1001a3cc <strcmp+0x178>
1001a3b8:       e0255002        eor     r5, r5, r2
1001a3bc:       e1550803        cmp     r5, r3, lsl #16
1001a3c0:       1a000006        bne     1001a3e0 <strcmp+0x18c>
1001a3c4:       e4902004        ldr     r2, [r0], #4
1001a3c8:       eafffff1        b       1001a394 <strcmp+0x140>
1001a3cc:       e1b0c80c        lsls    ip, ip, #16
1001a3d0:       1a00001b        bne     1001a444 <strcmp+0x1f0>
1001a3d4:       e1d130b0        ldrh    r3, [r1]
1001a3d8:       e1a05822        lsr     r5, r2, #16
1001a3dc:       ea00001b        b       1001a450 <strcmp+0x1fc>
1001a3e0:       e1a03803        lsl     r3, r3, #16
1001a3e4:       e1a05822        lsr     r5, r2, #16
1001a3e8:       e1a03823        lsr     r3, r3, #16
1001a3ec:       ea000017        b       1001a450 <strcmp+0x1fc>
1001a3f0:       e20250ff        and     r5, r2, #255    ; 0xff
1001a3f4:       e1550c23        cmp     r5, r3, lsr #24
1001a3f8:       e042c004        sub     ip, r2, r4
1001a3fc:       e1ccc002        bic     ip, ip, r2
1001a400:       1a000007        bne     1001a424 <strcmp+0x1d0>
1001a404:       e01cc384        ands    ip, ip, r4, lsl #7
1001a408:       04913004        ldreq   r3, [r1], #4
1001a40c:       1a000006        bne     1001a42c <strcmp+0x1d8>
1001a410:       e0255002        eor     r5, r5, r2
1001a414:       e1550403        cmp     r5, r3, lsl #8
1001a418:       1a000006        bne     1001a438 <strcmp+0x1e4>
1001a41c:       e4902004        ldr     r2, [r0], #4
1001a420:       eafffff2        b       1001a3f0 <strcmp+0x19c>
1001a424:       e1a03c23        lsr     r3, r3, #24
1001a428:       ea000008        b       1001a450 <strcmp+0x1fc>
1001a42c:       e31200ff        tst     r2, #255        ; 0xff
1001a430:       0a000003        beq     1001a444 <strcmp+0x1f0>
1001a434:       e4913004        ldr     r3, [r1], #4
1001a438:       e1a05422        lsr     r5, r2, #8
1001a43c:       e3c334ff        bic     r3, r3, #-16777216      ; 0xff000000
1001a440:       ea000002        b       1001a450 <strcmp+0x1fc>
1001a444:       e3a00000        mov     r0, #0
1001a448:       e8bd0030        pop     {r4, r5}
1001a44c:       e12fff1e        bx      lr
1001a450:       e20520ff        and     r2, r5, #255    ; 0xff
1001a454:       e20300ff        and     r0, r3, #255    ; 0xff
1001a458:       e3500001        cmp     r0, #1
1001a45c:       21500002        cmpcs   r0, r2
1001a460:       01a05425        lsreq   r5, r5, #8
1001a464:       01a03423        lsreq   r3, r3, #8
1001a468:       0afffff8        beq     1001a450 <strcmp+0x1fc>
1001a46c:       e0420000        sub     r0, r2, r0
1001a470:       e8bd0030        pop     {r4, r5}
1001a474:       e12fff1e        bx      lr

Wow!

Much different .dis

Strange. What do you see in build\command_line.elf.map?
Under "Archive member included to satisfy reference by file (symbol)" I see
c:/progra~1/raspbe~1/picosd~1.1/gcc-ar~1/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/lib/thumb/v6-m/nofp\libg.a(lib_a-strcmp.o)
                              CMakeFiles/command_line.dir/src/command.cpp.obj (strcmp)

Under "Linker script and memory map" there is
 .text          0x1001c920       0x88 c:/progra~1/raspbe~1/picosd~1.1/gcc-ar~1/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/lib/thumb/v6-m/nofp\libg.a(lib_a-strcmp.o)
                0x1001c920                strcmp

That file seems to be
-r-xr-xr-x 1 carlk carlk 1141972 Oct 17  2021 '/mnt/c/Program Files/Raspberry Pi/Pico SDK v1.5.1/gcc-arm-none-eabi/arm-none-eabi/lib/thumb/v6-m/nofp/libg.a'

If I run
readelf -a '/mnt/c/Program Files/Raspberry Pi/Pico SDK v1.5.1/gcc-arm-none-eabi/arm-none-eabi/lib/thumb/v6-m/nofp/libg.a'

I see
File: /mnt/c/Program Files/Raspberry Pi/Pico SDK v1.5.1/gcc-arm-none-eabi/arm-none-eabi/lib/thumb/v6-m/nofp/libg.a(lib_a-strcmp.o)
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          504 (bytes into file)
  Flags:                             0x5000000, Version5 EABI
  Size of this header:               52 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           40 (bytes)
  Number of section headers:         10
  Section header string table index: 9

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        00000000 000034 000088 00  AX  0   0  4
  [ 2] .data             PROGBITS        00000000 0000bc 000000 00  WA  0   0  1
  [ 3] .bss              NOBITS          00000000 0000bc 000000 00  WA  0   0  1
  [ 4] .debug_frame      PROGBITS        00000000 0000bc 000020 00      0   0  4
  [ 5] .rel.debug_frame  REL             00000000 000198 000010 08   I  7   4  4
  [ 6] .ARM.attributes   ARM_ATTRIBUTES  00000000 0000dc 00001b 00      0   0  1
  [ 7] .symtab           SYMTAB          00000000 0000f8 000090 10      8   8  4
  [ 8] .strtab           STRTAB          00000000 000188 00000e 00      0   0  1
  [ 9] .shstrtab         STRTAB          00000000 0001a8 00004d 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  y (purecode), p (processor specific)

There are no section groups in this file.

There are no program headers in this file.

There is no dynamic section in this file.

Relocation section '.rel.debug_frame' at offset 0x198 contains 2 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00000014  00000602 R_ARM_ABS32       00000000   .debug_frame
00000018  00000102 R_ARM_ABS32       00000000   .text

There are no unwind sections in this file.

Symbol table '.symtab' contains 9 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 00000000     0 SECTION LOCAL  DEFAULT    1
     2: 00000000     0 SECTION LOCAL  DEFAULT    2
     3: 00000000     0 SECTION LOCAL  DEFAULT    3
     4: 00000000     0 NOTYPE  LOCAL  DEFAULT    1 $t
     5: 00000084     0 NOTYPE  LOCAL  DEFAULT    1 $d
     6: 00000000     0 SECTION LOCAL  DEFAULT    4
     7: 00000000     0 SECTION LOCAL  DEFAULT    6
     8: 00000001   132 FUNC    GLOBAL DEFAULT    1 strcmp

No version information found in this file.
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6-M"
  Tag_CPU_arch: v6-M
  Tag_CPU_arch_profile: Microcontroller
  Tag_THUMB_ISA_use: Thumb-1

Okay I get:
Archive member included to satisfy reference by file (symbol)

/usr/lib64/gcc/arm-none-eabi/10/thumb/libgcc.a(_dvmd_tls.o)
                              CMakeFiles/command_line.dir/opt/pico-sdk/src/rp2_common/pico_divider/divider.S.obj (__aeabi_idiv0)

For strcmp():
 .text          0x000000001001a254      0x224 /usr/lib64/gcc/arm-none-eabi/10/../../../../arm-none-eabi/lib/thumb/libg.a(lib_a-strcmp.o)
                0x000000001001a254                strcmp

The file is:
-rw-r--r-- 2 root root 4940434 Jan  3  2023 /usr/arm-none-eabi/lib/thumb/libg.a

Whew, and finally the first 130 lines of nearly 10M of output:
File: /usr/arm-none-eabi/lib/thumb/libg.a(lib_a-_Exit.o)
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          1856 (bytes into file)
  Flags:                             0x5000000, Version5 EABI
  Size of this header:               52 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           40 (bytes)
  Number of section headers:         20
  Section header string table index: 19

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        00000000 000034 000008 00  AX  0   0  4
  [ 2] .rel.text         REL             00000000 0005a0 000008 08   I 17   1  4
  [ 3] .data             PROGBITS        00000000 00003c 000000 00  WA  0   0  1
  [ 4] .bss              NOBITS          00000000 00003c 000000 00  WA  0   0  1
  [ 5] .debug_info       PROGBITS        00000000 00003c 0000c5 00      0   0  1
  [ 6] .rel.debug_info   REL             00000000 0005a8 0000c0 08   I 17   5  4
  [ 7] .debug_abbrev     PROGBITS        00000000 000101 00008b 00      0   0  1
  [ 8] .debug_loclists   PROGBITS        00000000 00018c 00001e 00      0   0  1
  [ 9] .debug_aranges    PROGBITS        00000000 0001aa 000020 00      0   0  1
  [10] .rel.debug_a[...] REL             00000000 000668 000010 08   I 17   9  4
  [11] .debug_line       PROGBITS        00000000 0001ca 0000f1 00      0   0  1
  [12] .rel.debug_line   REL             00000000 000678 000008 08   I 17  11  4
  [13] .debug_str        PROGBITS        00000000 0002bb 000177 01  MS  0   0  1
  [14] .debug_frame      PROGBITS        00000000 000434 000028 00      0   0  4
  [15] .rel.debug_frame  REL             00000000 000680 000010 08   I 17  14  4
  [16] .ARM.attributes   ARM_ATTRIBUTES  00000000 00045c 00002a 00      0   0  1
  [17] .symtab           SYMTAB          00000000 000488 000100 10     18  14  4
  [18] .strtab           STRTAB          00000000 000588 000018 00      0   0  1
  [19] .shstrtab         STRTAB          00000000 000690 0000ad 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), y (purecode), p (processor specific)

There are no section groups in this file.

There are no program headers in this file.

There is no dynamic section in this file.

Relocation section '.rel.text' at offset 0x5a0 contains 1 entry:
 Offset     Info    Type            Sym.Value  Sym. Name
00000002  00000f0a R_ARM_THM_CALL    00000000   _exit

Relocation section '.rel.debug_info' at offset 0x5a8 contains 24 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00000008  00000702 R_ARM_ABS32       00000000   .debug_abbrev
0000000d  00000b02 R_ARM_ABS32       00000000   .debug_str
00000012  00000b02 R_ARM_ABS32       00000000   .debug_str
00000016  00000b02 R_ARM_ABS32       00000000   .debug_str
0000001a  00000202 R_ARM_ABS32       00000000   .text
00000022  00000a02 R_ARM_ABS32       00000000   .debug_line
00000029  00000b02 R_ARM_ABS32       00000000   .debug_str
00000037  00000b02 R_ARM_ABS32       00000000   .debug_str
0000003e  00000b02 R_ARM_ABS32       00000000   .debug_str
00000045  00000b02 R_ARM_ABS32       00000000   .debug_str
0000004c  00000b02 R_ARM_ABS32       00000000   .debug_str
00000053  00000b02 R_ARM_ABS32       00000000   .debug_str
0000005a  00000b02 R_ARM_ABS32       00000000   .debug_str
00000061  00000b02 R_ARM_ABS32       00000000   .debug_str
00000068  00000b02 R_ARM_ABS32       00000000   .debug_str
0000006f  00000b02 R_ARM_ABS32       00000000   .debug_str
00000076  00000b02 R_ARM_ABS32       00000000   .debug_str
0000007b  00000b02 R_ARM_ABS32       00000000   .debug_str
0000008d  00000b02 R_ARM_ABS32       00000000   .debug_str
00000094  00000202 R_ARM_ABS32       00000000   .text
0000009f  00000b02 R_ARM_ABS32       00000000   .debug_str
000000aa  00000802 R_ARM_ABS32       00000000   .debug_loclists
000000ae  00000802 R_ARM_ABS32       00000000   .debug_loclists
000000b3  00000202 R_ARM_ABS32       00000000   .text

Relocation section '.rel.debug_aranges' at offset 0x668 contains 2 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00000006  00000602 R_ARM_ABS32       00000000   .debug_info
00000010  00000202 R_ARM_ABS32       00000000   .text

Relocation section '.rel.debug_line' at offset 0x678 contains 1 entry:
 Offset     Info    Type            Sym.Value  Sym. Name
000000dc  00000202 R_ARM_ABS32       00000000   .text

Relocation section '.rel.debug_frame' at offset 0x680 contains 2 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
00000014  00000c02 R_ARM_ABS32       00000000   .debug_frame
00000018  00000202 R_ARM_ABS32       00000000   .text

There are no unwind sections in this file.

Symbol table '.symtab' contains 16 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 00000000     0 FILE    LOCAL  DEFAULT  ABS _Exit.c
     2: 00000000     0 SECTION LOCAL  DEFAULT    1 .text
     3: 00000000     0 SECTION LOCAL  DEFAULT    3 .data
     4: 00000000     0 SECTION LOCAL  DEFAULT    4 .bss
     5: 00000000     0 NOTYPE  LOCAL  DEFAULT    1 $t
     6: 00000000     0 SECTION LOCAL  DEFAULT    5 .debug_info
     7: 00000000     0 SECTION LOCAL  DEFAULT    7 .debug_abbrev
     8: 00000000     0 SECTION LOCAL  DEFAULT    8 .debug_loclists
     9: 00000000     0 SECTION LOCAL  DEFAULT    9 .debug_aranges
    10: 00000000     0 SECTION LOCAL  DEFAULT   11 .debug_line
    11: 00000000     0 SECTION LOCAL  DEFAULT   13 .debug_str
    12: 00000000     0 SECTION LOCAL  DEFAULT   14 .debug_frame
    13: 00000000     0 SECTION LOCAL  DEFAULT   16 .ARM.attributes
    14: 00000001     6 FUNC    GLOBAL DEFAULT    1 _Exit
    15: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND _exit

No version information found in this file.
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "4T"
  Tag_CPU_arch: v4T
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1

What are you thinking?

I also have another diagnostic idea. I'll write a quick program calling strcmp() and see if it falls over pointing to a bug in libg.a on Linux. Will fill in below with results.

Whew, and finally the first 130 lines of nearly 10M of output:
File: /usr/arm-none-eabi/lib/thumb/libg.a(lib_a-_Exit.o)

I would search for something like libg.a(lib_a-strcmp.o).
I think what you have there is for libg.a(lib_a-_Exit.o). But, assuming they were built with the same options, this looks suspect:
  Tag_CPU_arch: v4T

From https://github.com/ARM-software/abi-aa/blob/main/addenda32/addenda32.rst and https://developer.arm.com/documentation/dui0471/m/key-features-of-arm-architecture-versions/about-the-arm-architectures that looks like it's for an ARM922T processor, not Cortex-M0+.

What is decl_isr_bkpt isr_hardfault?

It's from pico-sdk\src\rp2_common\pico_standard_link\crt0.S.
It's the interrupt vector for the HardFault exception. In my latest code I'm overriding it (it is "weak") in src\crash.c.

Tag_CPU_arch: v4T

Oh, so if that is the issue, that's the packager for openSUSE who screwed the pooch with a processor mismatch. Shouldn't I be able to set a compile option like -march=.... that would set the target?
That would be from the cross-arm-none-eabi-newlib package, and there is only one version offered (other than for the pico nano). I'll poke around on the openSUSE list and find out what the bottom line is. They usually are very good on their builds.
I also have a build setup on Archlinux that I'll try. I'll compare the  Tag_CPU_arch there. I'll report back in the morning.

Tag_CPU_arch: v4T

Oh, so if that is the issue, that's the packager for openSUSE who screwed the pooch with a processor mismatch. Shouldn't I be able to set a compile option like -march=.... that would set the target?

I think v4T might apply to some full-blown Raspberry Pi. Just a guess.
You could take a look at build\CMakeFiles\CMakeOutput.log and search for something like Linking C executable. I'm not sure about reading these things, but I see]
COLLECT_GCC_OPTIONS='-mcpu=cortex-m0plus' '-mthumb' '-v' '-o' 'CMakeFiles/cmTC_6244a.dir/CMakeCXXCompilerABI.cpp.obj' '-c' '-mfloat-abi=soft' '-mlibarch=armv6s-m' '-march=armv6s-m'

Well, that is 100% right, and that's the same as what I have minus the -mfloat-abi=soft which simply isn't set. I have:
COLLECT_GCC_OPTIONS='-mcpu=cortex-m0plus' '-mthumb' '-v' '-std=gnu11' '-o' 'CMakeFiles/cmTC_cc2df.dir/CMakeCCompilerABI.c.obj' '-c' '-mlibarch=armv6s-m' '-march=armv6s-m'

I'll let you know what I get back from the SUSE devs -- they are usually some pretty bright folks.

Damn!! Just built the command_line example on Archlinux -- it works fine!!!
There is some issue somewhere with the openSUSE cross-arm-none-eabi package and I suspect it is an issue with the runtime and or the v4T issue (which I've left a question for the packager, but this guy has been slow in the past getting back to me)
Geeeze -- how much tail-chasing can one pooched library cause (I know the answer...). But all wasn't for not. Great learning of this package occurred, and of pico debug in generally. This library had some beneficial rearranging take place, so progress.
Thank you very much for your help and chasing this to ground. You can close this if you like, or leave it open for a few more days to give chance for a response from the openSUSE packager. Either way is fine.
It is a relief to at least confirm on Archlinux this isn't a generic Linux issue. Now the backspace key not backspacing in Linux looks generic. I'm running picocom and the cursor stays put -- but that isn't hard to fix on my end.
Whoop -- success. Not to find out what the packager has to say....

Hah! I fixed the cross-arm-none-eabi-newlib package. The issue was the options the original packager used to build it. He had included a number of options that were not used by Arch:
    --with-gnu-as \
    --with-gnu-ld \
    --enable-interwork \
    --enable-multilib \
    --enable-libstdcxx \
    --disable-libssp \

and had not included the options for C99 io formats, e.g.
    --enable-newlib-io-c99-formats \

Getting the options right and then rebuilding the package (about a 10 minute build on an i7 laptop with SSD), then installing and Presto, hooked the pico up and it's working!
> setrtc 15 12 23 2 1 40

> date
Fri Dec 15 02:01:45 2023
Day of year: 349

> mount 0:

> ls
Directory Listing: 0:/
System Volume Information [directory] [size=0]
bench.dat [writable file] [size=5242880]
numbers.txt [writable file] [size=61]
data [directory] [size=0]
cdef [directory] [size=0]
simple-SDIO.txt [writable file] [size=14]
simple.txt [writable file] [size=70]
minimal.txt [writable file] [size=28]

> info

Manufacturer ID: 0x1b
OEM ID: SMProduct: JB1RT
Revision: 3.0
Serial number: 0x605fed6b
Manufacturing date: 6/2021

SDHC/SDXC Card: hc_c_size: 61055
Sectors: 62521344
Capacity: 30528 MiB (32010 MB)
ERASE_BLK_EN: units of 512 bytes
SECTOR_SIZE (size of an erasable sector): 128 (65536 bytes)

  31248384 KiB (30516 MiB) total drive space.
  31241632 KiB (30509 MiB) available.

Filesystem type: FAT32
Volume base sector: 8192
FAT base sector: 9316
Root directory base sector (FAT12/16) or cluster (FAT32/exFAT): 2
Data base sector: 24576
FAT Cluster size ("allocation unit"): 64 sectors (32768 bytes)

Damn, what a good puzzle that I would rather have not had to put together...
Again, thank you for your help. This can be closed (and I'm sure we will have the opportunity to work on this library again :)

gcc takes a LOOONG time to build. (not to mention when it tanks after 8200 second packaging the RPM due to a files issue from the change in compile options...)
Quick question. I got a SDIO card. Had a bit of flakiness mounting the card and with card-detect. Does the SPI want the same external pullups when using SDIO mode?
After a bit of trouble, the second mount 0: worked fine, e.g.
> setrtc 15 12 23 20 55 35

> date
Fri Dec 15 20:55:40 2023
Day of year: 349

> mount 0:

> sd_card_detect:69: No SD card detected!
sd_card_detect:69: No SD card detected!

> sd_card_detect:69: No SD card detected!
sd_card_detect:69: No SD card detected!
sd_card_detect:69: No SD card detected!
...
> mount 0:

> ls
f_getcwd error: The volume has no work area (mount) (12)

> mount 0:

> ls
Directory Listing: 0:/
numbers.txt [writable file] [size=62]
data [directory] [size=0]
tmp [directory] [size=0]
System Volume Information [directory] [size=0]

Okay, but man does it work slick after it mounts, 3X the I/O:
> bench
Type is FAT32
Card size: 1305.67 GB (GB = 1E9 bytes)

Manufacturer ID: 0x27
OEM ID: PHProduct: SD32G
Revision: 6.0
Serial number: 0xb55b72da
Manufacturing date: 11/2022

FILE_SIZE_MB = 5
BUF_SIZE = 65536
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
5366.3,12383,12117,12193
5317.3,23302,11988,12276

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
6916.7,9704,9439,9463
6935.0,9696,9439,9451

Done

Wow!
Was the initial mount and ls issue likely related to lack of external pullup on this chip? Simple to add, but I wasn't sure if that would mess D0 up.
I managed to get both breakouts (SPI and SDIO) on the same breadboard.

Off topic, but with your familiarity with Linux, have you seen the Milk-V Duo? Sure, it costs more than a Pico (starting from $5), but it runs Linux and has SD card card support built in. I watched this video: https://youtu.be/z-GoLDQHYsE?si=6hNA00ZhO8qjEawS and was impressed.

I found my Stocking-Stuffer -- that looks really cool. My wife will shoot me, but I'm getting one.
On-topic. I got a SDIO breakout, actually got both the SPI and SDIO breakouts on the same breadboard as both Picos. After figuring out I still needed pullups on D0 and Card-Detect, it works great -- and 3X as fast as SPI!
> bench
Type is FAT32
Card size: 1305.67 GB (GB = 1E9 bytes)

Manufacturer ID: 0x27
OEM ID: PHProduct: SD32G
Revision: 6.0
Serial number: 0xb55b72da
Manufacturing date: 11/2022

FILE_SIZE_MB = 5
BUF_SIZE = 65536
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
5349.9,12388,12116,12189
5322.7,24512,11854,12295

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
6925.9,9698,9437,9462
6935.0,9698,9438,9451

Done

Your recent tweaks to the SDIO command_line example backend improved the write speeds from 4800 to 5300 (I had the code from old and new installed and tested both). I don't know if moving the memory around did it, or what, but you found a good combination.
I have finished the gcc build (8800 seconds) almost 2.5 hours of solid compilation (thankfully on the openSUSE buildservice servers). I'll rearrange my setup tomorrow and see if I can get the openSUSE install to play nicely.
I'll let you know how the Milk-V Duo goes :)

I still needed pullups on D0 and Card-Detect

I don't know why that would be. The pullup on DO is an MultiMediaCard (MMC) thing. (I don't think I've ever even owned an MMC).  SD cards use tristatable push-pull outputs instead of the MMC open collector bus, so a pullup shouldn't be needed. However, if you're pushing the baud rate, maybe you are seeing a line termination effect. Once you get up to a couple of tens of MHz, the transmission line effects become significant, and the slew rates of SD card outputs are extremely fast. After a lot of experimentation, I am now a big fan of serial source termination resistors. (And I wish my oscilloscope had a  bandwidth of more 25Mhz so I could see what is really happening!)
For Card-Detect, you should be able to use the RP2040 GPIO pad internal pull-up (See An instance of sd_card_t describes the configuration of one SD card socket.)

Only offered on Chinese websites from Hong Kong and those written in Chinese.

Bummer. I'm pretty sure apalrd still had some on his group buy at https://ko-fi.com/s/3a337eb05c when I looked. I should have jumped then. Kinda like when the Pico came out, and I ordered one for $4 on a whim. It wasn't long before I wanted a second one to use as a PicoProbe, but it was months before I could get one with any kind of normal purchase. I developed the first release of this project without a debugger.

All in good time. I was lucky with the Pico, my son was in EE and A&M and I bought 6 when then came out (and soldered all 240 pins...). I'll keep my eye on the Milk-V and see if they become more widely available. Though there I suspect supply-chain and automotive demand, etc.. will keep those tight for a while (ever).
I fully rebuilt the arm-none-eabi variants of gcc, binutils and newlib (gcc-12) mirroring the Archlinux build options. They now run the new command_line with the core0 move -- but bench hangs in the same spot that started this issue... Which is perplexing, but which I've also confirmed in spades isn't anything to do with this library.
This, I'm sure, will come down to a 1-off, strange, laptop USB interface or something similar. I have windows on the box and can dual-boot for a test. That will tell us if it is an openSUSE device config or a generic laptop hardware issue (old HP laptop with 17" display and i7 gen 2). It's almost to the point of needing an unreadable strace or a Wireshark type capture of what is going on when bench runs.
There is more of an interplay between the ARM toolchain and your host OS than I was aware of when you are hooked to your device. "semihosting" see StackOverflow Q  is what allows communication between your host and device. Also interesting while arm-none-eabi is used for compilation, your host compiler/linker is used to stitch the ARM toolchain together during link. So there are many places between the source file and seeing it run in a terminal where stray issues can creep in under the RADAR.
The dual-boot windows test will at least narrow the scope (a little).

Maybe the issue with the pullups stabilizing the SDIO communications is due to having both breakouts connected with all powered by the VSYS on the picoprobe.  But adding the pullups was a night-and-day difference. Before, you see the multiple mount commands and failures with ls. After, it is never missed a beat. I don't know how to explain that.

I'll keep my eye on the Milk-V and see if they become more widely available.

Heh, they are available at Wal*Mart.

bench hangs in the same spot that started this issue

Does anything happen if you hit Esc or Ctrl-C?

Maybe the issue with the pullups stabilizing the SDIO communications is due to having both breakouts connected with all powered by the VSYS on the picoprobe.

Nah, I commonly run 4 SD cards simultaneously, all powered by the PicoProbe (although I do have a bunch of decoupling caps.).

But adding the pullups was a night-and-day difference. Before, you see the multiple mount commands and failures with ls.

Did you try running at a lower baud rate? That might help if it's transmission line effects, as I suspect it is. What pull ups did you add? If it's only DO that's the issue then I think it must be transmission line effects. Is that line shared with the other SD breakout? Does it make any difference if you pop the other card out of the socket? I would expect a series resistor added as close as possible to the SD card in the DO line to perform better than the pull up. I'm getting baud_rate 31.5 MHz with this technology:

(I don't know if you can see it, but there is an old resistor there that I pulled out of my junk box.)
If you're running SPI, the DO line is probably the worst case, because it is driven by the SD card, and SD cards have extremely fast switching (since they are capable of such high speeds; much faster than the Pico can drive them). On the lines that are driven by the Pico, you can control the slew rate and drive current, but no matter what they will switch slower than lines driven by the SD card. However, I still like to add a series source termination resistor to the SCLK (CLK) line, because on that line, the SD card will detect every edge (up to a very high frequency, like 125 MHz), so if there's any ringing there will be trouble.

I'll keep my eye on the Milk-V and see if they become more widely available.
Heh, they are available at Wal*Mart.

I'm all over it -- and they can deliver by Jan. 9! (no stocking suffer...:()

bench hangs in the same spot that started this issue

Does anything happen if you hit Esc or Ctrl-C?

It simply doesn't complete on openSUSE. It runs, creates bench.dat apparently does both write and read tests and then never outputs any computations of what it did and just sits there. No prompt displayed and no "Done".
It's not hardlocked, all I need to do is hit [Enter] and the '>' prompt is displayed again and I can continue using the card normally. It is just really odd, and so far it only does this from my openSUSE Leap 15.4 laptop. Arch/Windows desktops are fine.
I've tried running at both 12 * 1000 * 1000 actual (10416666) and with 25 * 1000 * 1000 actual (20833332) no change.  I'm not sure what to make of it, but I suspect we have chased this down to the point where it is something odd in the way the USB attached to this laptop plays with the device -- but I stall have to do the dual-boot windows test...
There is another Oops that has popped up in the no-OS-FatFS-SD-SDIO-SPI-RPi-Pico command line build. The run_mount and run_unmount functions now warn of dereferencing a NULL pointer. The are concerned with sd_get_by_num() potentially returning NULL which goes back up the call-chain resulting in sd_card_p being NULL before derefernce.
I get:
[ 19%] Building CXX object CMakeFiles/command_line.dir/src/command.cpp.obj
/home/david/dev/arm/pico/tst/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/src/command.cpp: In function 'void run_mount(size_t, const char**)':
/home/david/dev/arm/pico/tst/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/src/command.cpp:288:24: warning: dereference of NULL 'sd_card_p' [CWE-476] [-Wanalyzer-null-dereference]
  288 |     sd_card_p->mounted = true;
      |     ~~~~~~~~~~~~~~~~~~~^~~~~~
  'void run_mount(size_t, const char**)': events 1-2
    |
    |  271 | static void run_mount(const size_t argc, const char *argv[]) {
    |      |             ^~~~~~~~~
    |      |             |
    |      |             (1) entry to 'run_mount'
    |  272 |     const char *arg = chk_dflt_log_drv(argc, argv);
    |      |                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    |      |                                       |
    |      |                                       (2) calling 'chk_dflt_log_drv' from 'run_mount'
<big snip>

and
/home/david/dev/arm/pico/tst/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/src/command.cpp: In function 'void run_unmount(size_t, const char**)':
/home/david/dev/arm/pico/tst/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/src/command.cpp:307:24: warning: dereference of NULL 'sd_card_p' [CWE-476] [-Wanalyzer-null-dereference]
  307 |     sd_card_p->mounted = false;
      |     ~~~~~~~~~~~~~~~~~~~^~~~~~~
  'void run_unmount(size_t, const char**)': events 1-2
    |
    |  290 | static void run_unmount(const size_t argc, const char *argv[]) {
    |      |             ^~~~~~~~~~~
    |      |             |
    |      |             (1) entry to 'run_unmount'
    |  291 |     const char *arg = chk_dflt_log_drv(argc, argv);
    |      |                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    |      |                                       |
    |      |                                       (2) calling 'chk_dflt_log_drv' from 'run_unmount'
<big snip>

I quieted them by making the else { in sd_get_by_num() return the address for the first element, e.g. return &sd_cards[0]; (which has to be defined)
Not sure how you want to handle that.
I'm going to check the other Wal-mart Milk-V offerings and see if I can't get one sooner!

Also on the wiring, no common lines. SPI  card is end card on GPIO 16-18 with card detect 22, the card closed to the pico is SDIO and it uses your default GPIO pins 2, 3, ...

All lines were custom cut to provide shortest direct distance to card (a bit of a pain, but worked out well)

Wiring looks good. It might be worth connecting a few more of the Pico's ground pins. And, as I mentioned above, it doesn't hurt to connect the 3V3 (out) of both Picos together and use that to power the SD cards.

It simply doesn't complete on openSUSE. It runs, creates bench.dat apparently does both write and read tests and then never outputs any computations of what it did and just sits there. No prompt displayed and no "Done".
It's not hardlocked, all I need to do is hit [Enter] and the '>' prompt is displayed again and I can continue using the card normally.

Is that true on both the SPI and SDIO -attached cards? Very strange. Maybe something is producing an escape character or something that upsets the terminal? You almost need to put a logic analyzer or something on it. Does the terminal program have any kind of tracing or diagnostics? I sometimes use tio, e.g. tio -m ODELBS /dev/ttyACM0. Seems to work OK.

On the series source termination for clk, not being a EE myself (Aero), would that be a pair of ~30 Ohm resistors? (or can I get by with a single resistor?) No clue how to actually calculate the impedance in a 1" 22Ga line. Wouldn't the same type source termination work for DO as well or would I need to adjust the size of the resistor?
Reading, source series is preferred over end-termination from an energy dissipation standpoint. The source only dissipates as transmission on transition, but end-termination dissipates the whole time the signal is high. So we want source termination, right? I was looking at Termination

On the series source termination for clk, not being a EE myself (Aero), would that be a pair of ~30 Ohm resistors? (or can I get by with a single resistor?)

I'm an old EE & CE. (Can't spell "geek" without EE!) You only need two resistors if it's a bidirectional line. The CLK is only driven at the Pico end (which would be the "source").

No clue how to actually calculate the impedance in a 1" 22Ga line. Wouldn't the same type source termination work for DO as well or would I need to adjust the size of the resistor?

Eh, rules of thumb are good enough here. A fat trace on a PCB might have a characteristic impedance down around 33 Ω; a flying wire is probably at around 100 Ω. But, in my experience, it is not critical. If you have a drawer full of 47 Ω resistors, close enough. (Ideally, of course, you would have access to an expensive oscilloscope which would make it easy to fine tune.)

Reading, source series is preferred over end-termination from an energy dissipation standpoint. The source only dissipates as transmission on transition, but end-termination dissipates the whole time the signal is high. So we want source termination, right?

Right. I think your pull-ups might be acting as a sort of end-termination. My intuitive understanding of series source termination is this:

A digital input (e.g., the Pico GPIO at the end of the DO line) is very high impedance: effectively an open circuit
When a signal propagates down a wire and hits an open end it reflects back and adds to whatever is coming towards that end
If you chose the resistor to be roughly the characteristic impedance of the line, a high pulse from the source will initially only raise the voltage on the line to 1/2 the voltage of the pulse
When the reflection from the open end is added to that you get the full pulse voltage
The destination end sees the pulse "instantaneously" go to the full voltage
When the reflection gets back to the series resistor, a lot of it is burned off so it doesn't bounce back again (inverted this time)

Without the resistor, the initial reflection raises the voltage to double what it should be until things settle and when it gets back to the source end it hits a very low impedance so it reflects again, but inverted now. The reflections go back and forth and result in "ringing". This can be pretty severe, and if the oscillations are big enough to cross logic levels and the input is sensitive enough, there will be problems. Clocks are especially vulnerable because data is clocked in or out on a clock edge. If you have extra edges, that's bad. Data lines are less of a problem because they typically have some time to settle to a level before the clock edge causes them to be sampled. Of course, the shorter the wire, the less time it takes to settle.

SDIO works attached to the laptop, SPI is the one that hangs -- good question. Yes, I have the full ttyACM0 device, so I should be able to debug there -- but have never been down that path before. Let me look at tio and see what I can do.

I wonder if there might be an interrupt conflict or something. It might be worth checking the Raspberry Pi docs. to see if they have any words about flow control or whatever for the USB virtual COM device. For some period of time, I guess it was before I had a PicoProbe, I used a cheapo USB to TTL adaptor to connect my terminal directly to pins 1 & 2 of the Pico under test. You can see it in this photo:

Got the Milk-V Duo (64MB) card and the Ethernet/USB Breakout on Amazon! Board and Breakout

I ordered a set, too. I even got the camera, since it was so cheap. I'm thinking if I like it I might buy this rig for my elderly father. He is also an old EE who has done some programming, and he spends a lot of time fiddling with cameras on his house in Boca Raton. He uses normal (cheap) security cameras, but they don't last long in the FL weather and salt. Not that the Duo would last any longer, but he might enjoy playing around with it. Of course, it would be too late for Christmas.

It might be worth checking the Raspberry Pi docs. to see if they have any words about flow control or whatever for the USB virtual COM device.

The pico has no flow control, so they stress disabling hardware flow control. I noted that PuTTY on windows defaults to XON/XOFF, so whether software flow control is set or not doesn't seem to make any difference. I've run picocom without any and with XON/XOFF set, no difference. Now I use the same USB/pico connector on each of the boxes, so that eliminates USB cabling.
My laptop and frankly all the boxes I have scattered around are about the same vintage 5-10 years old. Long ago computers got faster than I need them to be for what I do. Now we have played keeping up with the Jone's for the kids boxes as the gaming craze hit about 15 years ago. I've probably got 2-3 of every vintage of Nvidia card from the 8600GT (really old single-slot card) through the 2700GTX lying around....

Right. I think your pull-ups might be acting as a sort of end-termination.

That is what I thought as well, but at 5K on the SPI and 10K on the SDIO, they don't flow much, but maybe the ease the signal timing just enough to make things happy. No expensive oscilloscope or even a cheap trace-analyzer to play with, though on the windows side I have a couple of the freeware tools that will show a debug trace. I'll play with the pair of resistors (Amazon was great there too, got a bag of 600 resistors with 20 of every size from 10 Ohms to 1 MOhms, and grab bags of caps and transistors, etc...) I got into this helping my son through his EE/ESET degree, and it just kind of became a hobby, a new game of chess so to speak.
I almost got the camera as well, and after I make friends with the new card, I just may go back and get it :)

No expensive oscilloscope or even a cheap trace-analyzer to play with

Get yourself something like this for $10 or so: Comidox 1Set USB Logic Analyzer Device Set USB Cable 24MHz 8CH 24MHz 8 Channel UART IIC SPI Debug for Arduino ARM FPGA M100 Hot. It works great with PulseView - sigrok on Linux. I used that combination for years until my Linux laptop's motherboard died. Unfortunately, PulseView - sigrok on Windows is terrible, so I sprung for this: DreamSourceLab DSLogic Plus USB-Based Logic Analyzer with 400MHz Sampling Rate, 256Mbits Memory, USB 2.0 Interface, 16 ChannelsDreamSourceLab DSLogic Plus USB-Based Logic Analyzer with 400MHz Sampling Rate, 256Mbits Memory, USB 2.0 Interface, 16 Channels, which is very nice. Both have decoders for dozens of protocols including SD and SD/SPI.

I'll play with the pair of resistors

While you're at it, double up your grounds and power supplies. I try to avoid solderless breadboards since I have wasted so much time trying to debug a problem that went away when I added a (seemingly redundant) ground wire.
You could also try slowing everything down:

Set the SPI baud_rate down to 1000000 [side note: you need to do this if you want to look at the SPI bus with the cheapo logic analyzer]
Uncomment

    // {"set_sys_clock_48mhz", run_set_sys_clock_48mhz,
    //  "set_sys_clock_48mhz:\n"
    //  " Set the system clock to 48MHz"},

in src\command.cpp and use that, or just stick
    set_sys_clock_48mhz();
    setup_default_uart();

into main.

Amazon was great there too, got a bag of 600 resistors with 20 of every size from 10 Ohms to 1 MOhms, and grab bags of caps and transistors, etc...

I should get one of those. Way back in the '70s I was in our school's A/V and had access to a Fluke digital multimeter. I bought a grab bag of resistors from someone like Digikey for very little $, with the caveat that they were all unmarked. I spent hours with the Fluke measuring the resistors and putting masking tape tags on them. I even roped a couple of friends into helping me. I'm still using that collection today, but it is getting really sparse in some ranges, and the masking tape is dried out and nasty.

One more thing you can slow down: you could add PICO_DEFAULT_UART_BAUD_RATE=9600 (or something) to your compile_definitions in CMakeLists.txt.

I'll have to try. In all our changes, I have now lost SPI completely. I can load an old executable and the code runs fine over SPI, but I cannot compile and load a new executable using SPI at all.
Result:
> date
Mon Dec 18 16:28:20 2023
Day of year: 352
> mount 0:
f_mount error: The physical drive cannot work (3)

Nothing has changed on the hardware, so there is a new game a-foot.
Also, you are missing a time.h include:
In file included from /home/david/dev/arm/pico/tst/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/main.cpp:7:
/home/david/dev/arm/pico/tst/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/include/crash.h:84:5: error: 'time_t' does not name a type
   84 |     time_t timestamp;
      |     ^~~~~~
/home/david/dev/arm/pico/tst/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/include/crash.h:1:1: note: 'time_t' is defined in header '<ctime>'; did you forget to '#include <ctime>'?
  +++ |+#include <ctime>
    1 | /* crash.h
In file included from /home/david/dev/arm/pico/tst/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/examples/command_line/main.cpp:10:
/home/david/dev/arm/pico/tst/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/src/include/rtc.h:20:8: error: 'time_t' does not name a type
   20 | extern time_t epochtime;
      |        ^~~~~~
/home/david/dev/arm/pico/tst/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/src/include/rtc.h:1:1: note: 'time_t' is defined in header '<ctime>'; did you forget to '#include <ctime>'?
  +++ |+#include <ctime>
    1 | /* rtc.h
make[2]: *** [CMakeFiles/command_line.dir/build.make:90: CMakeFiles/command_line.dir/main.cpp.obj] Error 1
make[1]: *** [CMakeFiles/Makefile2:1528: CMakeFiles/command_line.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

Oh the irony, "to err is human, but to really foul things up requires a computer...."
I just so happen to have two of the HP-8760w laptops, so I booted the other to win10 and fired up PuTTY, and plugged the picos in with the same software as before, and ...
> bench
Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0x1b
OEM ID: SMProduct: JB1RT
Revision: 3.0
Serial number: 0x605fed6b
Manufacturing date: 6/2021

FILE_SIZE_MB = 5
BUF_SIZE = 65536
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1863.8,37946,34676,35060
1871.1,38123,33566,34947

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1996.5,33210,32711,32834
1996.5,33210,32710,32829

Done

SPI and bench work like a champ. You just can't make this stuff up. That is conclusive that whatever the issue that bench is running into, it's happening only under openSUSE on this one laptop. The final test will be to reboot this laptop into win10 and see if the same issue occurs... Will report the final straw back.

Also, you are missing a time.h include:

While I certainly agree, for some reason when I do a clean build, I get "Build finished with exit code 0" and no warnings. Must be some difference in the prior include files, or maybe different compiler options, or ...?

bench hangs at the same point

Did you try lowering the PICO_DEFAULT_UART_BAUD_RATE? (Obviously, you'd have to change your terminal's baud rate setting, too.)

Hold on, I'll do that now, earlier I was dismissing the baud difference, as in the spi_t .baud_rate because that communication was fine, but the terminal baud rate may be doing something odd? Even though the windows baud for PuTTY was set at 115200 N81, maybe picocom and minicom and tio don't like it. Will report the results back.

I have made a pretty substantial change to how the DMA interrupts are handled. There were definitely issues with the old way.
I doubt that it's related, but ya never know. If you want to give it a try, checkout branch "dev".

Will checkout "dev" and report back on the SPI bench issue on Linux. Question, should add_compile_definitions(USE_LED=1) work for SDIO? It works fine for SPI. I'll double check, may have a stale cache issue, but I thought I deleted CMakeCache.txt. But generally, should it work for both?

Question, should add_compile_definitions(USE_LED=1) work for SDIO?

No, I never implemented that. Haven't even thought about where it would go. Most people seem to hate it. I like it because I'm old enough to remember when PCs had a big red drive activity LED on the front panel. It has been helpful to me in tuning an application because it makes it easy to see the drive utilization at a glance.
Strangely enough, the Pico-W used the old LED GPIO as an SPI line to control the wireless module. USE_LED=1 was originally the default, and the conflict caused one poor guy to waste a lot of time. At that point, I defaulted it to be disabled.

Well..... dev code works great -- but, just like all earlier versions did for SPI on this laptop. I am convinced at this point this has nothing to do with this library and must be due to something odd openSUSE is doing with the serial device. I have compared stty settings between openSUSE and Archlinux, and they are identical, so the terminal setups are the same.
Of course there is always another "middle-layer" involved, and in this case udev manages permissions for access to the serial device. My last best hope is that there is something wonky in the udev setup under openSUSE. I'll run that down, but I have a real hard time thinking that there could be something that just effect the output of bench and nothing else. (I doubt there will be any difference between the openSUSE and Archlinux udev-rule for ttyACM0, but it is worth confirming)
More likely there is a odd buffering issue under openSUSE somewhere or a difference in the how the kernel versions handle the kernel/userspace syscalls. (openSUSE 15.4 is running the Linux 5.14.21 kernel, while Archlinux is always on the bleeding-edge with 6.6.7) Also worth checking, but honestly, over the past 15 years there really haven't been any changes I would expect to impact anything to do with a serial port. Last major change in how the kernel handled things was way back in the 3.26 version. Fundamental changes since then have been mild.
Good improvements, Flash usage is back down to 7.99%.
Unplugged the board from the laptop and plugged it into the Archlinux box, about 8' away, ssh'ed into the box and fired up picocom, set the clock on the pico and:
> bench
Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0x1b
OEM ID: SMProduct: JB1RT
Revision: 3.0
Serial number: 0x605fed6b
Manufacturing date: 6/2021

FILE_SIZE_MB = 5
BUF_SIZE = 65536
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1867.8,38125,33645,34990
1855.9,35430,34959,35213

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
1939.7,34043,33632,33780
1940.4,34043,33631,33773

Done

Chuckling and shaking head at the same time ....

Maybe you could make a test program that just uses printf to generate various test patterns of characters at various rates. You could even cut and paste the bench output as one of the test patterns, maybe printing it in the same sized chunks that bench does.
Anyway, thanks for testing the "dev" branch.

We are presently investigating if the issue is due to kernel options used by openSUSE. This laptop makes things a bit difficult as it has a built-in Nvidia card, (390.xx driver version) so testing the suggested 6.6.7 kernel on openSUSE means patching the Nvidia driver for the 6.6 kernel and rebuilding the driver. Now we seem stuck in a loop waiting on the openSUSE buildservice scheduler to pickup the new packages and either publish to my buildservice repo or give me a failed indication. Hurry up -- and wait...

Here is an interesting one on the dev branch. I configured both cards the SPI and SDIO to work at the same time rather than just isolating them in hw_config.c. When doing so, I gave the SPI card the .pcName = "1:" and the SDIO .pcName = "0:".
However, when I loaded the program and did mount 0: -- it mounted the SPI card and mount 1: mounted the SDIO card. The relevant part of hw_config.c is:
static sd_card_t sd_cards[] = {  // One for each SD card
    {   // sd_cards[0]: Socket sd0
        (See http://elm-chan.org/fsw/ff/doc/filename.html#vol) */
        .pcName = "1:",
        .type = SD_IF_SPI,
        .spi_if_p = &spi_ifs[0],  // Pointer to the SPI interface driving this card
        .use_card_detect = true,
        .card_detect_gpio = 22,
        .card_detected_true = 1, // What the GPIO read returns when a card is
        .card_detect_use_pull = true,
        .card_detect_pull_hi = true
    },
    {   // sd_cards[0]: Socket sd0
        .pcName = "0:",  // Name used to mount device
        .type = SD_IF_SDIO,
        .sdio_if_p = &sdio_ifs[0],  // Pointer to the SPI interface driving this card
        .use_card_detect = true,
        .card_detect_gpio = 9,
        .card_detected_true = 1, // What the GPIO read returns when a card is
        .card_detect_use_pull = true,
        .card_detect_pull_hi = true
    }
};

But they are happily working together under the swapped .pcName strings:
> sc 23 12 23 20 22 0
> date
Sat Dec 23 20:22:05 2023
Day of year: 357
> mount 0:
> ls
Directory Listing: 0:/
System Volume Information [directory] [size=0]
bench.dat [writable file] [size=5242880]
numbers.txt [writable file] [size=61]
data [directory] [size=0]
cdef [directory] [size=0]
simple-SDIO.txt [writable file] [size=14]
simple.txt [writable file] [size=70]
minimal.txt [writable file] [size=42]
> mount 1:
> chdrive 1:
> ls
Directory Listing: 1:/
numbers.txt [writable file] [size=62]
data [directory] [size=0]
tmp [directory] [size=0]
System Volume Information [directory] [size=0]
bench.dat [writable file] [size=5242880]
>

(and yes, 0: really is the SPI accessed card, even though it has the file simple-SDIO.txt on it from an earlier test)
> bench 1:
Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)

Manufacturer ID: 0x1b
OEM ID: SMProduct: JB1RT
Revision: 3.0
Serial number: 0x605fed6b
Manufacturing date: 6/2021

FILE_SIZE_MB = 5
BUF_SIZE = 65536
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
5736.2,46179,10728,11370
6026.3,11150,10853,10866

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
6990.5,9636,9367,9385
6990.5,9639,9367,9378

Done

This isn't a critical issue, just thought I would pass it along, and if we don't talk until after the holidays, have a Merry x-mas and a happy new year!

Yeah, if you're using the default configuration, FatFS associates each logical drive with the physical drive of the same drive number. (See Volume Management.) So, mount 0: will mount physical drive 0, which in this case is the 0th element of the sd_cards[] array, regardless of the pcName. It's a bit confusing having the pcName attribute in this case, since it is more or less irrelevant.  However, other configurations are possible. For example, if you specified FF_STR_VOLUME_ID=1 and
const char* VolumeStr[FF_VOLUMES] = {"sd1", "sd0"};
you could get behavior more like what you're expecting. (I don't think
const char* VolumeStr[FF_VOLUMES] = {"1", "0"};
will work, because numeric drive number is always valid regardless of  FF_STR_VOLUME_ID, and will be interpreted as physical drive number.) When I added the pcName attribute was more interested in the case of FF_STR_VOLUME_ID=2, which gives Unix-like behavior, where everything is in a single hierarchical directory tree. This is more compatible with FreeRTOS-FAT-CLI-for-RPi-Pico for portability of applications (and test cases).
Happy Christmas to you, too!

I worked with .pcName long enough to be comfortable with it just being a name and not critical to the drive numbering. I didn't wade far enough to pull out the effect of FF_STR_VOLUME_ID. Decided not to tempt fate too much with a house full of grown up kids home.
I did play a bit further with SDIO baud rate and dramatically improved throughput to over 9 MBs write and 11 MBs read, e.g.
> bench 1:
Type is FAT32
Card size: 1305.67 GB (GB = 1E9 bytes)

Manufacturer ID: 0x27
OEM ID: PHProduct: SD32G
Revision: 6.0
Serial number: 0xb55b72da
Manufacturing date: 11/2022

FILE_SIZE_MB = 5
BUF_SIZE = 65536
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
8752.7,37341,6744,7424
9329.0,7259,6995,7007

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
11323.7,5995,5762,5781
11372.8,5994,5761,5771

Done

Quite pleased and surprised it plays so well as 25MHz with my wonky 10K resistor setup.
Also investigating adapting the USE_LED for SDIO with a PIO triggering on the SDIO CMD pin and then inverting the signal to the LED. PIO has this neat feature where you can JMP PIN which  reduces the PIO assembly to:
.program blink
.wrap_target

start:              ; LED off label
  set pins 0        ; turn LED off
hi:                 ; LED on label
  jmp pin, start    ; check gpio pin, if hi, jmp to start
  set pins 1        ; pin low, turn LED on
  jmp hi            ; jmp to hi

.wrap

(I'm quite sure there are shorter ways to do it, but where I am in making friends with PIO, I was happy with a three-instruction loop for checking for change and toggling the LED)
No FIFOs or PUSH/PULL involved. Then the only config needed is to set the jump pin with either sm_config_set_jmp_pin() before calling pio_sm_init() or after using the CMSIS directly, e.g. pio->sm[sm].execctrl |= (SDIO_CMD_PIN << PIO_SM0_EXECCTRL_JMP_PIN_LSB); Still working on where to implement it in the code. I'll update if I can craft a good solution.
Have a happy New Year!

You got me thinking, and I came to the conclusion that pcName was redundant and confusing, so I decided to remove it from the configuration. Instead, it is constructed when sd_init_driver() is called, and written to sd_card_t.state.drive_prefix. There is a new function to get the drive prefix: char const *sd_get_drive_prefix(sd_card_t *sd_card_p).
It turned out to be trickier than I expected, and I'm not perfectly satisfied with the results. The biggest issue is that sd_card_t.state.drive_prefix is not valid until sd_init_driver() is called. In some use cases, the first time the SD driver was touched was when f_mount was called, so f_mount indirectly called sd_init_driver(). The catch-22 is that the signature of f_mount is this:
FRESULT f_mount (
  FATFS*       fs,    /* [IN] Filesystem object */
  const TCHAR* path,  /* [IN] Logical drive number */
  BYTE         opt    /* [IN] Initialization option */
);

so you need to supply the drive prefix in the path parameter, and I often found it convenient to use pcName for that. However, sd_card_t.state.drive_prefix isn't valid until sd_init_driver() is called, and sd_init_driver() is called from f_mount. A solution is to explicitly call sd_init_driver() before f_mount, but that is a new requirement.
Anyway, I have pushed the changes to https://github.com/carlk3/no-OS-FatFS-SD-SDIO-SPI-RPi-Pico/tree/dev.
You have a happy New Year too!

I got somewhat familiar with PIO when I was adopting the ZuluSCSI driver to this library. However, it has been a while. The next big challenge I see for this project is support for 1-bit SDIO. 1-bit SDIO would probably be about as fast as SPI (in other words, a lot slower than 4-bit SDIO!), but it would have the advantage of the smallest hardware footprint of all. For SPI, you need:

SCLK; clock
MOSI (or COPI, or "peripheral's Data Input [DI]")
MISO (or CIPO, or "peripheral's Data Output [DO]")
SS (or CS); select

as well as an RP2040 SPI hardware component. Pin assignments are limited.
For 1-bit SDIO you need:

CLK; clock
CMD; command
DAT0; data (bidirectional)

and the pins could be anywhere.
There is an implementation in pico_sd_card, but it is half-baked. I'm not sure whether it would be easier to start from that or to start from the 4-bit driver code (or just start over!). In any case, it's going to take some PIO.

I will check out the dev branch and the pcName change and see if anything pops up. Yes a check-or-the-egg situation. I think of it a "convenience with side-effects" :). While I'm thinking on it, there is an issue with how bench reports card-size for SDIO (it is fine for SPI). The issue is in the use of CSD for sector count in src/sd_driver/SDIO/sd_card_sdio.c.
Currently to get the sector count CSD_sectors(sd_card_p->CSD) is used (line 536 in sd_sdio_sectorCount(sd_card_t *sd_card_p)).  This is also set on line 129 with sd_card_p->sectors = CSD_sectors(sd_card_p->CSD). I haven't got the number completely worked out, but it looks like a 4K/100 error somewhere. Meaning there is a block size (or something) of 4096 that gets divided by 100 and the sector count for SDIO bench is off by a factor of that amount (40.96), e.g.
> bench 1:
Type is FAT32
Card size: 1305.67 GB (GB = 1E9 bytes)
                   ^^^^^^

(it's a 32G card)
Using SPI, the report is correct:
> bench 0:
Type is FAT32
Card size: 32.01 GB (GB = 1E9 bytes)
                  ^^^^

Hit a "need more research" block when I hit the CMD9_SEND_CSD parameter in rp2040_sdio_command_R2(sd_card_p, CMD9_SEND_CSD, ... on line 124 above.... Likely will save that puzzle piece for the New Year! Have a good one!

there is an issue with how bench reports card-size for SDIO

I had an erroneous byte swap in CSD_sectors.

Many changes in command.c (yes, I reverted to C...)

I'm not sure whether or not it's true on this platform, but on many the main must be C++ if you want to call any C++ code anywhere in the program. Anyway, some users of this library are using C++, and they ran into various problems; e.g.: I was using "this" as a parameter name, and that's a reserved keyword in C++. So, I try to remember to do a little testing with C++. (I used  C++ in my day job for about 12 years, but for the last six it's been almost all C. I have been surprised at how out of date my knowledge of C++ now is. It's a moving target, and one almost has to make a career out of it to be any good at it.)

You can use C++ from C any time and vice-versa, the header guards allow the C in C++, but as long as the linker will compile the code, it doesn't care whether its main.c or main.cpp.

I wish that would work on STM32. I've got a couple of projects in STM32CubeIDE and the C++ support really sucks. [Actually, a lot of it really sucks.] They provide a menu option to "Convert to C++", but if you change anything in the hardware configurator it generates a main.c (while leaving your old main.cpp in place, just to add to the confusion). But calling C++ won't work there unless the main is C++ or you jump through some more hoops (even though it is still the ARM GCC toolchain). So, I decided to stick to pure C there, even though every once in a while I come across something that would be a lot easier in C++.

Got MILK?

It has just arrived. I haven't even opened it yet, but I am looking forward to it. I kinda gave up on repairing my 10 year old Dell XPS 15 and went ahead and got a new Acer Swift Edge, so soon I will have a Linux development environment again, in addition to Windows, for the first time in over a year. But, it will probably take me a while to get everything set up.

When you plug you milkV board in (no need to solder pins yet), you can connect via RNDIS (ethernet over USB -- horribly insecure MS Win95 hack), but it will get you up and running. Just start at milkV Overview and then Getting Started will walk you through getting up and running.
You can flash the default image (1.07) and not worry about building the buildroot-sdk (when you clone the duo-examples that will download the needed duo-sdk (1.2.0). Being able to ssh to the milkV-duo is a godsend for developing. Things are so much easier. I've got it all soldered up and am about to drop it into the ethernet/USB IO Board and hook it up to the network.  Then it's just another computer on the net (like a full-Pi) to develop on.
Only downside is the Linux it runs is busybox - limited, but fine for initial purposes. You may find the pinout strikingly similar to the Pico -- what a coincidence! Amazing how that happens.
(I swear to God the pin-headers are getting smaller or my eyes are going. I ended up having to "Star Wars" solder most of it -- "using the Force Luke..." because I damn sure couldn't see the individual pins clearly... but came out just fine. Soldering-by-braille really does work! )  Note: Linux To Try Again To Disable All RNDIS Protocol Drivers so you may find your rndis modules blacklisted in your Linux distro -- they were in openSUSE, and I had fits until I figured that out. Another good puzzle.

[Perhaps we should take this discussion offline, or to the MilkV community forum or something. My email is carlk3@gmail.com.]
I stumbled upon this Pico project today: Pi Pico Rx. That got me thinking about what kind of Software Defined Radio (SDR) could built around the Milk-V Duo. He uses the Pico PIO, so that part would have to be redesigned. Maybe the extra processor in the Duo could be used instead? For the Digital Signal Processing (DSP) side of things, the Duo lacks floating point hardware, but with the right algorithms I guess that is not a problem. On the other hand, it has the vector extension, which could be really useful!
Meanwhile, back on this project, I'm on another side track. I ordered an eMMC module and I'm going to see what I can do with that. eMMC and SD protocols are very similar -- to the point that many eMMC modules come with a trivial adaptor that allows them to be plugged into a microSD socket. Attaching one to the Pico that way is not very interesting, but eMMC has other bus modes:

Two possibilities stand out:

By using the 8 bit interface instead of the 4 bit SD interface transfer rate, the transfer rate could potentially be doubled.
Dual Data Rate (DDR)! That clocks the data on the rising AND falling edges of the clock. SD has DDR, but only for 1.8 V signaling. eMMC has a DDR mode that uses 3.3 V signaling.

Between the two, it might be possible to multiply the transfer rate by up to four. A write speed of 25 MB/s and reading at 50 MB/s might be within reach. High quality video recording on the Pico?
I think the 8-bit interface would be easy to implement. DDR, however, might take some tricky PIO coding.