Skip to content

Update Lombiq packages to v13 (major) #64

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
May 4, 2025
Merged

Update Lombiq packages to v13 (major) #64

merged 4 commits into from
May 4, 2025

Conversation

LombiqBot
Copy link
Member

@LombiqBot LombiqBot commented May 4, 2025 •
edited
Loading

This PR contains the following updates:

Package Type Update Change
Lombiq.Tests.UI nuget major 12.1.1-alpha.15.osoe-1064 -> 13.0.0
Lombiq.Tests.UI.AppExtensions nuget major 12.1.1-alpha.15.osoe-1064 -> 13.0.0
Lombiq.Tests.UI.Shortcuts (source) nuget major 12.1.1-alpha.15.osoe-1064 -> 13.0.0

@Lombiq/dotnest-sites-renovate-notification-recipients

Release Notes

Lombiq/UI-Testing-Toolbox (Lombiq.Tests.UI)

v13.0.0

Highlights

This is quite a large and impactful release. We've added a lot of improvements to make your tests rock-solid, but there are also a couple of breaking changes. See the details below, but here's a summary:

  • Browser log management now uses the cross-browser Selenium BiDirectional API, allowing accessing the log of every browser, not just Chrome.
  • Upgraded xUnit to v3, which is not only necessary for maintenance, but also brings proper cancellation of tests. No more dangling browser or driver processes after a test cancellation!
  • You can test not just sending e-mails with a local SMTP server, but also receiving e-mails.
  • Fixing reliability issues that Chrome v134 brought. This Chrome version broke a huge number of tests in intricate ways, most possibly due to its more aggressive resource management.
  • To avoid random bugs due to browsers updating on their own, all browsers are now pinned to a specific version, and automatically managed behind the scenes, in all environments (every OS, locally, in a CI environment...).
  • Tests using Elasticsearch can be run in parallel without any special configuration (this isn't trivial, since they mustn't use the same Elasticsearch index) and they'll automatically wait for the initial indexing to complete, not to try to search too early.
  • And a lot of smaller improvements and bug fixes!
Breaking changes in browser log management

The browser log, i.e. what you also see in the developer console, is useful to pinpoint unhandled JavaScript exceptions, network errors, and other client-side issues. This is what in Selenium can be accessed via IWebDriver.Manage().Logs.GetLog(LogType.Browser). In the UI Testing Toolbox it was accessible via various other ways, and was also automatically asserted after every page load.

In this release, how you can access the browser log, and what objects represent the log, changed. This was necessary because the previous version actually only supported Chrome and Edge, it just silently failed under other browsers (notably Firefox). With a new Selenium version, it now fails with an exception (that's why we actually noticed it).

To overcome this, all browsers now use the new Selenium BiDirectional API to access the logs.

What changed
  • UITestContext.HistoricBrowserLog is now called CumulativeBrowserLog to better reflect that it stores all browser log entries since the start of the test. Instead of containing OpenQA.Selenium.LogEntry objects, it contains OpenQA.Selenium.BiDi.Modules.Log.Entry ones. These have a slightly different interface, but work much the same: Most possibly, you'll only need to change references to LogEntry.Message to Entry.Text, and LogEntry.Level to Entry.Level, which uses a different, but very similar, enum. For the same reason, ClearHistoricBrowserLog() is now called ClearCumulativeBrowserLog() but behaves the same.
  • Similar to the response log, see below, the browser log now has a filter, UITestContext.BrowserLogFilter, to determine what gets logged in the first place. If you need to ignore expected log entries (since a non-empty log fails the test by default), we recommend adjusting the filter instead of the assertion. The samples contain examples of this.
  • OrchardCoreUITestExecutorConfiguration.AssertBrowserLog now works with OpenQA.Selenium.BiDi.Modules.Log.Entry objects too, as do all related helpers.
  • UITestContext.UpdateHistoricBrowserLogAsync() was removed. It's not necessary anymore since the logs are now updated not explicitly, but with events, whenever a new entry is added.
  • UITestContext.AssertCurrentBrowserLogAsync() and the IWebDriver extension GetAndEmptyBrowserLog() were also removed for the same reason.
  • Unexpected response codes, like 404 responses, aren't logged to the browser log anymore. Thus, LogEntry.IsNotFoundLogEntry() was removed since it can't apply to Entry objects. Instead, check out UITestContext.CumulativeResponseLog and UITestContext.ResponseLogFilter. This new behavior is demonstrated in the samples.
  • BiDi keeps a tab open for itself, what you mustn't close (if you do, the browser session will also close). This shouldn't cause an issue in most cases, but check if your tests assume that only their tabs are open (like if you use context.SwitchToLastWindow() or context.SwitchToLastWindow()).
How to adapt
  • The most important issues will show up as build errors. Fix them following the guidelines above.
  • Check if you have assertion code for the browser log containing 404 or other network errors. These won't show up in the browser log anymore, so you'll have to adjust your assertions to use the response log, see above.
  • Running your test suite will make any further issues apparent.
  • We also encourage you to explore the BiDi APIs, since they provide functionality not seen in Selenium before:
var biDi = await context.Scope.Driver.AsBiDiAsync();
// Check out biDi.Network, for example.
Upgraded xUnit to v3 with breaking changes

This version of the UI Testing Toolbox uses v3 of xUnit, which brings many updates. However, it's also a breaking version, requiring you to adapt your test projects, see the official guide.

Migrating UI test projects consuming the UI Testing Toolbox should be simpler, though, with you requiring to do roughly the following steps:

  1. Update the xunit.runner.visualstudio package reference to latest (currently 3.0.2).
  2. Convert test projects to executables, see docs (this will most possibly only need you to add <OutputType>Exe</OutputType> to the first PropertyGroup in the test project's csproj). These are all the projects with xunit.runner.visualstudio references.
  3. In all test projects, change xunit package references to xunit.v3 with the latest version (currently 2.0.0). If a test project lacks such a reference then add it (currently <PackageReference Include="xunit.v3" Version="2.0.0" />). Only test projects should reference xunit, but they must reference it, otherwise tests won't be discovered.
  4. Also update the Microsoft.NET.Test.Sdk references in the test projects to latest (currently 17.13.0) too while we're at it, or add it if it's missing (currently <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.13.0" />).
  5. ITestOutputHelper is now in the Xunit namespace instead of Xunit.Abstractions. Fix the namespace imports in UITestBase classes first and then anywhere else; they'll show up as build errors.
  6. Clean up unused namespace imports. In Visual Studio, you can do this by right-clicking on the solution -> Analyze and Code Cleanup -> Run Code Cleanup (Profile 1).
  7. Run the tests to check if everything is still OK. If tests don't show up in the Visual Studio Test Explorer, or show up but don't start if you try to run them, check out the Tests pane of the Output Window for clues. Confirm that the Web project doesn't contain any xUnit references (even transitively, like by accidentally referencing UI test projects) but all test projects reference xunit.v3. You might need to do a recursive git clean to be sure to start with a clean slate.
  8. Verify in CI that the tests are actually discovered and all run there too (the CI build being green is not enough, it might miss no tests being discovered).
Complete changelog
New Contributors

Full Changelog: Lombiq/UI-Testing-Toolbox@v12.0.0...v13.0.0

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about these updates again.

  • If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

@github-actions GitHub Actions
Copy link

github-actions bot commented May 4, 2025

This pull request has merge conflicts. Please resolve those before requesting a review.

@LombiqBot LombiqBot force-pushed the renovate/major-lombiq-packages branch from e70b9da to 84567d2 Compare May 4, 2025 11:36
@DemeSzabolcs
Migrating to Lombiq.Tests.UI 13
7280b73
@DemeSzabolcs
Merge remote-tracking branch 'origin/renovate/major-lombiq-packages' …
3a65eb0 
…into renovate/major-lombiq-packages

# Conflicts:
#	test/DotNest.Core.SDK.Tests.UI/DotNest.Core.SDK.Tests.UI.csproj
@DemeSzabolcs DemeSzabolcs merged commit fc431d7 into dev May 4, 2025
5 checks passed
@LombiqBot LombiqBot deleted the renovate/major-lombiq-packages branch May 11, 2025 04:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
dependencies renovate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@LombiqBot @DemeSzabolcs @renovate-bot