yorickr/plezy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf(test): scale test concurrency and stop re-onboarding every Maestro flow

Browse Source 
The Dart suite spent 77% of its cost compiling one isolate per test file
while `flutter test` used half the cores, and every Maestro flow replayed
a full Jellyfin onboarding before its first real assertion.

- Add scripts/run_tests.sh, which runs `flutter test` with -j set to the
  cores the process may actually use instead of the ncpu/2 default.
  Measured on 8 cores: 190s -> 136s; -j 12 regresses to 165s, so it scales
  to the core count rather than hard-coding one. CI and CONTRIBUTING use it.
  A cgroup v2 quota, a cgroup v1 quota, and the cpuset/affinity nproc
  reports can each be the binding limit independently, so the detector
  takes the smallest; trusting whichever it found first would oversubscribe
  4x on a container holding an 8-CPU quota while pinned to 2. Covered by
  scripts/test_run_tests.py, which the ci_guard_checks.sh glob picks up.
- Add .maestro/subflows/ensure_onboarded.yaml: cold-start the app and only
  onboard when no session is stored. Flows that just need a signed-in Home
  use it; 02_onboarding_home, 08_logout, 09_download_offline_playback and
  the profile regressions keep clearing state. 59s -> 16s per flow.
- Guard onboarding's two optional taps behind visibility checks. A missed
  `optional: true` tap still runs the full element search, costing 3.0s
  and 7.8s per onboarding to find nothing.
- Disable device animation scales in run_maestro.py, restored by the
  existing cleanup path. CI's emulator got this from the runner flag;
  physical devices never did.
- Shorten the watch_together setup-timeout replacement from 500ms to the
  10ms the same file already proves sufficient, and shorten the retry
  backoff at the one site that missed it: 8.04s -> 1.59s of execution.
- Make the LAN discovery waits deadline-based and resend the beacon while
  polling. Loopback UDP drops datagrams under load, which timed out a
  wait that could never be satisfied; this was the suite's one flaky test.
- Fix 08_logout, which searched for "Logout" and "Are you sure you want to
  logout?" after both strings became "Log out". The flow had been failing
  and aborting the suite before 09 ever ran.

flutter test 190s -> 131s. Maestro's Android suite 621s -> 385s across the
eight flows the baseline reached, and now runs all nine green.
This commit is contained in:
edde746
2026-07-27 03:59:27 +02:00
parent 8e1904ddee
commit 41a2e996e1
18 changed files with 462 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/services/companion_remote_lan_discovery_service_test.dart
+96 -22
View File
@@ -14,11 +14,17 @@ void main() {
final listener = await _DiscoveryListener.start([context]);
try {
await listener.sendBeacon(context: context, ips: const ['192.0.2.10']);
await _waitFor(() => listener.emissions.length == 1);
await listener.sendBeaconUntil(
condition: () => listener.emissions.length == 1,
context: context,
ips: const ['192.0.2.10'],
);
await listener.sendBeacon(context: context, ips: const ['192.0.2.30', '10.0.0.30']);
await _waitFor(() => listener.emissions.length == 2);
await listener.sendBeaconUntil(
condition: () => listener.emissions.length == 2,
context: context,
ips: const ['192.0.2.30', '10.0.0.30'],
);
final hosts = listener.emissions.last;
expect(hosts, hasLength(1));
@@ -38,12 +44,21 @@ void main() {
final listener = await _DiscoveryListener.start([context]);
try {
await listener.sendBeacon(context: context, platform: 'macOS', ips: const ['192.0.2.40', '10.0.0.40']);
await _waitFor(() => listener.emissions.length == 1);
await listener.sendBeaconUntil(
condition: () => listener.emissions.length == 1,
context: context,
platform: 'macOS',
ips: const ['192.0.2.40', '10.0.0.40'],
);
// Reordered IPs only: must be absorbed without publishing an emission.
await listener.sendBeacon(context: context, platform: 'macOS', ips: const ['10.0.0.40', '192.0.2.40']);
await listener.sendBeacon(context: context, platform: 'Android', ips: const ['192.0.2.40', '10.0.0.40']);
await _waitFor(() => listener.emissions.any((hosts) => hosts.single.platform == 'Android'));
await listener.sendBeaconUntil(
condition: () => listener.emissions.any((hosts) => hosts.single.platform == 'Android'),
context: context,
platform: 'Android',
ips: const ['192.0.2.40', '10.0.0.40'],
);
expect(listener.emissions, hasLength(2));
final hosts = listener.emissions.last;
@@ -63,12 +78,21 @@ void main() {
final listener = await _DiscoveryListener.start([firstContext, secondContext]);
try {
await listener.sendBeacon(context: firstContext, name: 'Living Room', ips: const ['192.0.2.50']);
await _waitFor(() => listener.emissions.length == 1);
await listener.sendBeaconUntil(
condition: () => listener.emissions.length == 1,
context: firstContext,
name: 'Living Room',
ips: const ['192.0.2.50'],
);
// Context churn only: must be absorbed without publishing an emission.
await listener.sendBeacon(context: secondContext, name: 'Living Room', ips: const ['192.0.2.50']);
await listener.sendBeacon(context: secondContext, name: 'Living Room TV', ips: const ['192.0.2.50']);
await _waitFor(() => listener.emissions.any((hosts) => hosts.single.name == 'Living Room TV'));
await listener.sendBeaconUntil(
condition: () => listener.emissions.any((hosts) => hosts.single.name == 'Living Room TV'),
context: secondContext,
name: 'Living Room TV',
ips: const ['192.0.2.50'],
);
expect(listener.emissions, hasLength(2));
final hosts = listener.emissions.last;
@@ -96,6 +120,14 @@ RemoteAuthContext _authContext({required String id, required List<int> discovery
);
}
// Loopback UDP needs a real poll: there is no seam to drive the socket's
// receive path from fake time. Keep the interval short so a wait costs about
// what the round trip costs, and bound every wait by a deadline rather than an
// attempt count so the budget stays fixed if the interval changes.
const _pollInterval = Duration(milliseconds: 2);
const _resendInterval = Duration(milliseconds: 250);
const _waitBudget = Duration(seconds: 5);
class _DiscoveryListener {
_DiscoveryListener._({
required this.service,
@@ -129,13 +161,13 @@ class _DiscoveryListener {
}
}
Future<void> sendBeacon({
List<int> _encodeBeacon({
required RemoteAuthContext context,
required List<String> ips,
String name = 'Living Room',
String platform = 'macOS',
int port = 52100,
}) async {
required String name,
required String platform,
required int port,
}) {
const version = 1;
final auth = RemoteAuthService.instance;
final homeHash = auth.computeDiscoveryTag(context.discoveryKey);
@@ -149,7 +181,7 @@ class _DiscoveryListener {
port: port,
ips: ips,
);
final packet = utf8.encode(
return utf8.encode(
jsonEncode({
'app': 'plezy',
'v': version,
@@ -162,14 +194,55 @@ class _DiscoveryListener {
'hmac': hmac,
}),
);
}
for (var attempt = 0; attempt < 100; attempt++) {
Future<void> _transmit(List<int> packet) async {
final deadline = DateTime.now().add(_waitBudget);
while (DateTime.now().isBefore(deadline)) {
if (sender.send(packet, InternetAddress.loopbackIPv4, service.discoveryPort) == packet.length) return;
await Future<void>.delayed(const Duration(milliseconds: 20));
await Future<void>.delayed(_pollInterval);
}
fail('Timed out sending LAN discovery beacon');
}
Future<void> sendBeacon({
required RemoteAuthContext context,
required List<String> ips,
String name = 'Living Room',
String platform = 'macOS',
int port = 52100,
}) {
return _transmit(_encodeBeacon(context: context, ips: ips, name: name, platform: platform, port: port));
}
/// Sends a beacon repeatedly until [condition] holds.
///
/// Loopback UDP drops datagrams when the host is loaded, and a lost beacon
/// leaves a plain [_waitFor] polling for an emission that can never arrive —
/// the observed flake in this suite. Re-sending is safe because the payload
/// is byte-identical and the service suppresses beacons that carry no
/// change, so only the first datagram to land can publish an emission.
Future<void> sendBeaconUntil({
required bool Function() condition,
required RemoteAuthContext context,
required List<String> ips,
String name = 'Living Room',
String platform = 'macOS',
int port = 52100,
}) async {
final packet = _encodeBeacon(context: context, ips: ips, name: name, platform: platform, port: port);
final deadline = DateTime.now().add(_waitBudget);
while (DateTime.now().isBefore(deadline)) {
await _transmit(packet);
final resendAt = DateTime.now().add(_resendInterval);
while (DateTime.now().isBefore(resendAt)) {
if (condition()) return;
await Future<void>.delayed(_pollInterval);
}
}
fail('Timed out waiting for LAN discovery behavior');
}
Future<void> close() async {
sender.close();
await subscription.cancel();
@@ -178,9 +251,10 @@ class _DiscoveryListener {
}
Future<void> _waitFor(bool Function() condition) async {
for (var attempt = 0; attempt < 100; attempt++) {
final deadline = DateTime.now().add(_waitBudget);
while (DateTime.now().isBefore(deadline)) {
if (condition()) return;
await Future<void>.delayed(const Duration(milliseconds: 20));
await Future<void>.delayed(_pollInterval);
}
fail('Timed out waiting for LAN discovery behavior');
}
test/watch_together/watch_together_peer_service_test.dart
+2 -6
View File
@@ -30,7 +30,7 @@ Future<T> _withShortenedTimer<T>({
Future<T> _withSetupTimersShortened<T>(Future<T> Function() body) {
return _withShortenedTimer(
original: const Duration(seconds: 10),
replacement: const Duration(milliseconds: 500),
replacement: const Duration(milliseconds: 10),
body: () => _withShortenedTimer(
original: const Duration(milliseconds: 250),
replacement: const Duration(milliseconds: 1),
@@ -531,11 +531,7 @@ void main() {
final timeoutService = serviceFor(timeoutRelay);
await expectLater(
_withShortenedTimer(
original: const Duration(seconds: 10),
replacement: const Duration(milliseconds: 10),
body: () => timeoutService.createSession(sessionId: 'slow1'),
),
_withSetupTimersShortened(() => timeoutService.createSession(sessionId: 'slow1')),
throwsA(
isA<PeerError>()
.having((error) => error.type, 'type', PeerErrorType.timeout)