iOS testbed fails when the dir ~/Library/Developer/XCTestDevices is empty

[joerick]

Bug report

Bug description:

iOS testbed run fails when the dir ~/Library/Developer/XCTestDevices is empty.

As seen in pypa/cibuildwheel#2443 .

The dir ~/Library/Developer/XCTestDevices is empty on machines that are clean, e.g. CI images.

The traceback is:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/private/var/folders/x8/34szswsn0zn1ktnx8k1v16qr0000gn/T/cibw-run-pztsyqge/cp313-ios_arm64_iphonesimulator/testbed/__main__.py", line 548, in <module>
    main()
    ~~~~^^
  File "/private/var/folders/x8/34szswsn0zn1ktnx8k1v16qr0000gn/T/cibw-run-pztsyqge/cp313-ios_arm64_iphonesimulator/testbed/__main__.py", line 530, in main
    asyncio.run(
    ~~~~~~~~~~~^
        run_testbed(
        ^^^^^^^^^^^^
    ...<3 lines>...
        )
        ^
    )
    ^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 719, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/private/var/folders/x8/34szswsn0zn1ktnx8k1v16qr0000gn/T/cibw-run-pztsyqge/cp313-ios_arm64_iphonesimulator/testbed/__main__.py", line 411, in run_testbed
    simulator = await select_simulator_device()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/x8/34szswsn0zn1ktnx8k1v16qr0000gn/T/cibw-run-pztsyqge/cp313-ios_arm64_iphonesimulator/testbed/__main__.py", line 129, in select_simulator_device
    raw_json = await async_check_output(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
        "xcrun", "simctl", "--set", "testing", "list", "-j"
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/private/var/folders/x8/34szswsn0zn1ktnx8k1v16qr0000gn/T/cibw-run-pztsyqge/cp313-ios_arm64_iphonesimulator/testbed/__main__.py", line 118, in async_check_output
    raise subprocess.CalledProcessError(
    ...<4 lines>...
    )
subprocess.CalledProcessError: Command '('xcrun', 'simctl', '--set', 'testing', 'list', '-j')' returned non-zero exit status 1.

On a machine with empty ~/Library/Developer/XCTestDevices the above command returns:

$ xcrun simctl --set testing list -j
  
Using Parallel Testing Device Clones Device Set: '/Users/runner/Library/Developer/XCTestDevices'
Provided set path does not exist: /Users/runner/Library/Developer/XCTestDevices

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-135101: When choosing the default simulator device, don't use simctl --set testing #135102
• [3.14] gh-135101: When choosing the default simulator device, don't use simctl --set testing (GH-135102) #135113
• [3.13] gh-135101: When choosing the default simulator device, don't use simctl --set testing (GH-135102) #135114

[freakboy3742]

Thanks for the report. This is one of those bugs that is infuriatingly difficult to reproduce, because as soon as you've run the simulator successfully, you never see the problem again. Thankfully it looks like this one might be reproducible on GitHub Actions.

I've previously fixed an issue related to this - #130388 - however, when I added support for autodetecting the default simulator in #133132, it looks like I've missed the corresponding error handling. Your proposed solution of removing the use of the testing set for device discovery makes sense; I'll check a couple of other edge cases to be sure.

[joerick]

Thanks for merging!

This is one of those bugs that is infuriatingly difficult to reproduce, because as soon as you've run the simulator successfully, you never see the problem again.

I found that I could reproduce it by doing-

mv ~/Library/Developer/XCTestDevices ~/Library/Developer/XCTestDevices_bkup

Then I'd play around with it, then delete the dir and restore the bkup once done.

[freakboy3742]

I found that I could reproduce it by doing-...

Good to know; my concern is whether that's actually "clean" from the perspective of Xcode - is the directory the only artefact that Xcode is relying on? It may well be... but it's impossible to know for sure.