Xcode 16.3 issue with advance build settings

[Dlemex]

The Spezi Template Application was using a version prior to 4.7.2, and I encountered a SourcePackages directory not found error. I worked with a member of their project to resolve the error. After updating to your latest release, 4.7.2, the error persisted with my installation, but it did not occur with their installation. I investigated further after reviewing your source, and I determined that the issue was related to my Xcode setting that stored my build products in folders within the project folder instead of a unique DerivedData folder.

The project would compile without error with Xcode 16.2, but reproducibly with Xcode 16.3. The detailed issue is available at their project: StanfordSpezi/SpeziTemplateApplication#103

In that report, you can see the advanced settings I was using with Xcode. I had a Build folder at the project's top level containing separate folders for intermediate and product files. All other Xcode locations settings for DerivedData were the default values.

I would be happy to help you more if you'd like.

[FelixHerrmann]

Hi @Dlemex, thanks for reaching out! Before I read through all of it would you be able to quickly integrate the package with a specific branch? I have generalized the fixes from 4.7.1 & 4.7.2 in https://github.com/FelixHerrmann/swift-package-list/tree/improvement/generalize-plugin-source-packages-location-fix and wonder if this might fix your issue already.

[Dlemex]

@FelixHerrmann I will give it a try later this afternoon.

[Dlemex]

@FelixHerrmann I have good news and bad news. The bad news is that the changes in that branch do not eliminate the problem. The good news is that this is a problem within Xcode. In 16.3, when I set the custom Build folder, both the branch and 4.7.2 worked without error. I was frustrated, so I cleared the custom build setting in 16.3 and returned to 16.2 and set the custom build setting there. Then, in 16.3, I could reproduce the problem with the branch and 4.7.2.

Finally, I returned to 16.2 and verified that the improvement branch worked without issues.

[Dlemex]

@FelixHerrmann This problem is so frustrating. Now that it is failing, it is failing constantly in 16.3. My solution of returning to the default setting does not fix the problem.

I have even tried resetting to the default settings using 16.2 and clearing DerivedData and Build folders, but 16.3 continues to fail with both the branch and 4.7.2.

[FelixHerrmann]

Can you show me the error message?

[Dlemex]

The message is reported below, with the configuration set to use a unique folder for building. I am including a screenshot of the actual contents of the DerivedData. It clearly shows it is pointing to an unexpected location.

Showing All Messages
working directory:
/Users/Shared/Projects/XcodeProjects/Spezi_10_3
tool mapping:
swift-package-list: /${BUILD_DIR}/${CONFIGURATION}/swift-package-list
tool search paths:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
/Applications/Xcode.app/Contents/Developer/usr/bin
/bin
/sbin
/usr/bin
/usr/sbin

SourcePackages directory not found from /Users/dlemex/Library/Developer/Xcode/DerivedData/Build/Intermediates/BuildToolPluginIntermediates/TemplateApplication.output/TemplateApplication/SwiftPackageListPlugin
SourcePackages directory not found from /Users/dlemex/Library/Developer/Xcode/DerivedData/Build/Intermediates/BuildToolPluginIntermediates/TemplateApplication.output/TemplateApplication/SwiftPackageListPlugin

[FelixHerrmann]

Thanks @Dlemex, it is clear why it fails. Just wonder how you were able to get this folder structure, is it with the unique option or with the configuration from the previous screenshot? I will play around with those location settings and see if I can produce this scenario on my side (and what this means for the SourcePackages location).

[Dlemex]

Hi @FelixHerrmann Today, I did several things to try to return to a working configuration, including using defaults to delete all Xcode settings and reverting the project folder changes. Xcode still creates the erroneous Build folder. Building a different project without PlugIns does not make that folder. When I build the project with Xcode 16.2, that folder is still created, but your plugin is happy. I honestly believe that Xcode 16.3 is unstable. I appreciate your investigation.

[FelixHerrmann]

Xcode 16.3 is clearly problematic, they def. have changed something bigger. Your custom config made me thinking tho, it is basically not guaranteed that the plugin work directory is "meeting" the SourcePackages its way up. Maybe I should remove the --custom-source-packages flag entirely from the plugin and invoke it the default way.

Btw the normal CLI tool should work for you, at least I would assume that from your derived data screenshot; a custom run script should work therefore as well.

[sebastianricaldoni]

Hi @FelixHerrmann and @Dlemex ! We can confirm this is also happening to us inside our CI/CD pipeline (fastlane inside GH actions).
I can also recreate the issue locally by running:

xcodebuild -resolvePackageDependencies -scheme ExampleProject -project ./ExampleProject.xcodeproj -derivedDataPath .derivedData

set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme ExampleProject -project ./ExampleProject.xcodeproj -derivedDataPath .derivedData -destination 'platform=iOS Simulator,id=FE3CA45C-EDFF-4946-9097-850318F0D888' -resultBundlePath ./ExampleProject.xcresult -parallel-testing-worker-count 1 -maximum-concurrent-test-simulator-destinations 1 -enableCodeCoverage YES -skipPackagePluginValidation -skipMacroValidation test | tee ExampleProject.log | xcbeautify

Testing failed:
	SourcePackages directory not found in /Users/seb/projects/celteeka/ExampleProject/.derivedData/Build/Intermediates.noindex/BuildToolPluginIntermediates/ExampleProject.output/ExampleProject/SwiftPackageListPlugin
	Testing cancelled because the build failed.

** TEST FAILED **


The following build commands failed:
	Apply build tool plug-in “SwiftLintBuildToolPlugin” to target “ExampleProjectUITests” in project “ExampleProject”
	Apply build tool plug-in “SwiftLintBuildToolPlugin” to target “ExampleProjectTests” in project “ExampleProject”
	Apply build tool plug-in “SwiftLintBuildToolPlugin” to target “ExampleProject” in project “ExampleProject”
	Apply build tool plug-in “SwiftPackageListPlugin” to target “ExampleProject” in project “ExampleProject”
	
	Testing project ExampleProject with scheme ExampleProject
(6 failures)

I can confirm that the folder / directory DOES indeed exist in the mentioned location. We'll continue adding more information as we make progress.

[FelixHerrmann]

Thank you @sebastianricaldoni. One thing you should def. try is executing the CLI tool itself (install it via homebrew), at least that should be possible because you can specify the SourcePackages location directly.

[sebastianricaldoni]

Thanks @FelixHerrmann for your quick reply and support.
Running the CLI works great. I also tried using 4.7.2 as well as the branch you mentioned above but neither worked in GH Actions. Looking a bit deeper the error we are getting is:

dyld[11909]: Library not loaded: @rpath/SwiftPackageList.framework/Versions/A/SwiftPackageList

  Reason: tried: '/usr/lib/swift/SwiftPackageList.framework/Versions/A/SwiftPackageList' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/swift/SwiftPackageList.framework/Versions/A/SwiftPackageList' (no such file), '/usr/lib/swift/SwiftPackageList.framework/Versions/A/SwiftPackageList' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/swift/SwiftPackageList.framework/Versions/A/SwiftPackageList' (no such file)

This is happening when the building process reaches "Run custom shell script 'SwiftPackageListPlugin'

I'll continue with my research and post my updates here in case I find any solution.
My take is that the sandboxed Shell script generated by xcodebuild is not being a tad more restrictive with file permissions and perhaps the plugin needs to whitelist some other folder(s) in the outputFiles return value
Thanks!

[FelixHerrmann]

That is an interesting new error @sebastianricaldoni. Feel free to open a separate issue if it has nothing to do with the changed build directory location.

[FelixHerrmann]

I have pushed another possible solution into https://github.com/FelixHerrmann/swift-package-list/tree/tmp/xcode-16-3-testing @sebastianricaldoni.

You can now configure the --custom-source-packages-path in the configuration like in the following example. Keep in mind that normal builds (in Xcode directly) will not work but in combination with #143 (comment) this might be the most flexible solution.

{
    "project" : {
        "outputType" : "stdout",
        "requiresLicense" : false,
        "customSourcePackagesPath" : ".derivedData/SourcePackages"
    }
}

Let me know if you had a chance to test this.

[sebastianricaldoni]

Hey @FelixHerrmann . Sorry I didn't get back to you earlier. I tried the branch but still the same results, at least for me. I wasn't able to continue digging deeper and get myself more acquainted with Swift Package management specially in the context of fastlane and GH Actions. I'll keep you posted with any findings or fixes.
Thanks!

[FelixHerrmann]

Thanks for checking @sebastianricaldoni. So you are getting the dyld error?

[PSchmiedmayer]

@FelixHerrmann Any plans to merge https://github.com/FelixHerrmann/swift-package-list/tree/improvement/generalize-plugin-source-packages-location-fix into main? We have been pointing to this branch in e.g. StanfordBDHG/MyHeartCounts-iOS#8 and it seems to pass the CI there.

Not sure about the use case of @sebastianricaldoni & @Dlemex. Also adding @lukaskollmer & @NikolaiMadlener here for some additional context. Thank you for all the work of maintaining this package @FelixHerrmann!

[FelixHerrmann]

@PSchmiedmayer yes I can create a release later today, I didn't know my generalized solution is actually covering another case. There is still the interest to control the plugin from the invocation side but I didn't had the time yet to fully think through that.

[PSchmiedmayer]

@FelixHerrmann Sounds good to me; thank you for all the support and maintaining the package!