# Flutter DeviceLab
DeviceLab is a physical lab that tests Flutter on real devices.
This package contains the code for the test framework and tests. More generally
the tests are referred to as "tasks" in the API, but since we primarily use it
for testing, this document refers to them as "tests".
Current statuses for the devicelab are available at
<https://flutter-dashboard.appspot.com/#/build>. See [dashboard user
guide](https://github.com/flutter/cocoon/blob/main/dashboard/USER_GUIDE.md)
for information on using the dashboards.
## Table of Contents
* [How the DeviceLab runs tests](#how-the-devicelab-runs-tests)
* [Running tests locally](#running-tests-locally)
* [Writing tests](#writing-tests)
* [Adding tests to continuous
integration](#adding-tests-to-continuous-integration)
* [Adding tests to presubmit](#adding-tests-to-presubmit)
* [Migrating to build and test model](#migrating-to-build-and-test-model)
## How the DeviceLab runs tests
DeviceLab tests are run against physical devices in Flutter's lab (the
"DeviceLab").
Tasks specify the type of device they are to run on (`linux_android`, `mac_ios`,
`mac_android`, `windows_android`, etc). When a device in the lab is free, it
will pickup tasks that need to be completed.
1. If the task succeeds, the test runner reports the success and uploads its
performance metrics to Flutter's infrastructure. Not all tasks record
performance metrics.
2. If task fails, an auto rerun happens. Whenever the last run succeeds, the
task will be reported as a success. For this case, a flake will be flagged and
populated to the test result.
3. If the task fails in all reruns, the test runner reports the failure to
Flutter's infrastructure and no performance metrics are collected
## Running tests locally
Do make sure your tests pass locally before deploying to the CI environment.
Below is a handful of commands that run tests in a similar way to how the
CI environment runs them. These commands are also useful when you need to
reproduce a CI test failure locally.
### Prerequisites
You must set the `ANDROID_SDK_ROOT` environment variable to run
tests on Android. If you have a local build of the Flutter engine, then you have
a copy of the Android SDK at `.../engine/src/third_party/android_tools/sdk`.
You can find where your Android SDK is using `flutter doctor -v`.
### Warnings
Running the devicelab will do things to your environment.
Notably, it will start and stop Gradle, for instance.
### Running specific tests
To run a test, use option `-t` (`--task`):
```sh
# from the .../flutter/dev/devicelab directory
../../bin/cache/dart-sdk/bin/dart bin/test_runner.dart test -t {NAME_OF_TEST}
```
Where `NAME_OR_PATH_OF_TEST` is the name of a task, which is a file's
basename in `bin/tasks`. Example: `complex_layout__start_up`.
To run multiple tests, repeat option `-t` (`--task`) multiple times:
```sh
../../bin/cache/dart-sdk/bin/dart bin/run.dart -t test1 -t test2 -t test3
```
### Running tests against a local engine build
To run device lab tests against a local engine build, pass the appropriate
flags to `bin/run.dart`:
```sh
../../bin/cache/dart-sdk/bin/dart bin/run.dart --task=[some_task] \
--local-engine-src-path=[path_to_local]/engine/src \
--local-engine=[local_engine_architecture] \
--local-engine-host=[local_engine_host_architecture]
```
An example of a local engine architecture is `android_debug_unopt_x86` and
an example of a local engine host architecture is `host_debug_unopt`.
### Running an A/B test for engine changes
You can run an A/B test that compares the performance of the default engine
against a local engine build. The test runs the same benchmark a specified
number of times against both engines, then outputs a tab-separated spreadsheet
with the results and stores them in a JSON file for future reference. The
results can be copied to a Google Spreadsheet for further inspection and the
JSON file can be reprocessed with the `summarize.dart` command for more detailed
output.
Example:
```sh
../../bin/cache/dart-sdk/bin/dart bin/run.dart --ab=10 \
--local-engine=host_debug_unopt \
--local-engine-host=host_debug_unopt \
-t bin/tasks/web_benchmarks_canvaskit.dart
```
The `--ab=10` tells the runner to run an A/B test 10 times.
`--local-engine=host_debug_unopt` tells the A/B test to use the
`host_debug_unopt` engine build. `--local-engine-host=host_debug_unopt` uses
the same engine build to run the `frontend_server` (in this example).
`--local-engine` is required for A/B test.
`--ab-result-file=filename` can be used to provide an alternate location to
output the JSON results file (defaults to `ABresults#.json`). A single `#`
character can be used to indicate where to insert a serial number if a file with
that name already exists, otherwise, the file will be overwritten.
A/B can run exactly one task. Multiple tasks are not supported.
Example output:
```text
Score Average A (noise) Average B (noise) Speed-up
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 2900.20 (8.44%) 2426.70 (8.94%) 1.20x
bench_card_infinite_scroll.canvaskit.totalUiFrame.average 4964.00 (6.29%) 4098.00 (8.03%) 1.21x
draw_rect.canvaskit.windowRenderDuration.average 1959.45 (16.56%) 2286.65 (0.61%) 0.86x
draw_rect.canvaskit.sceneBuildDuration.average 1969.45 (16.37%) 2294.90 (0.58%) 0.86x
draw_rect.canvaskit.drawFrameDuration.average 5335.20 (17.59%) 6437.60 (0.59%) 0.83x
draw_rect.canvaskit.totalUiFrame.average 6832.00 (13.16%) 7932.00 (0.34%) 0.86x
```
The output contains averages and noises for each score. More importantly, it
contains the speed-up value, i.e. how much _faster_ is the local engine than
the default engine. Values less than 1.0 indicate a slow-down. For example,
0.5x means the local engine is twice as slow as the default engine, and 2.0x
means it's twice as fast. Higher is better.
Summarize tool example:
```sh
../../bin/cache/dart-sdk/bin/dart bin/summarize.dart --[no-]tsv-table --[no-]raw-summary \
ABresults.json ABresults1.json ABresults2.json ...
```
`--[no-]tsv-table` tells the tool to print the summary in a table with tabs for
easy spreadsheet entry. (defaults to on)
`--[no-]raw-summary` tells the tool to print all per-run data collected by the
A/B test formatted with tabs for easy spreadsheet entry. (defaults to on)
Multiple trailing filenames can be specified and each such results file will be
processed in turn.
## Reproducing broken builds locally
To reproduce the breakage locally `git checkout` the corresponding Flutter
revision. Note the name of the test that failed. In the example above the
failing test is `flutter_gallery__transition_perf`. This name can be passed to
the `run.dart` command. For example:
```sh
../../bin/cache/dart-sdk/bin/dart bin/run.dart -t flutter_gallery__transition_perf
```
## Writing tests
A test is a simple Dart program that lives under `bin/tasks` and uses
`package:flutter_devicelab/framework/framework.dart` to define and run a _task_.
Example:
```dart
import 'dart:async';
import 'package:flutter_devicelab/framework/framework.dart';
Future<void> main() async {
await task(() async {
... do something interesting ...
// Aggregate results into a JSONable Map structure.
Map<String, dynamic> testResults = ...;
// Report success.
return new TaskResult.success(testResults);
// Or you can also report a failure.
return new TaskResult.failure('Something went wrong!');
});
}
```
Only one `task` is permitted per program. However, that task can run any number
of tests internally. A task has a name. It succeeds and fails independently of
other tasks, and is reported to the dashboard independently of other tasks.
A task runs in its own standalone Dart VM and reports results via Dart VM
service protocol. This ensures that tasks do not interfere with each other and
lets the CI system time out and clean up tasks that get stuck.
## Adding tests to continuous integration
Host only tests should be added
没有合适的资源?快使用搜索试试~ 我知道了~
资源推荐
资源详情
资源评论
收起资源包目录
Flutter 移动开发框架 v3.19.2.zip (2000个子文件)
win32_window.cpp 8KB
win32_window.cpp 8KB
win32_window.cpp 8KB
win32_window.cpp 8KB
win32_window.cpp 7KB
win32_window.cpp 7KB
win32_window.cpp 7KB
flutter_window.cpp 6KB
flutter_window.cpp 5KB
flutter_window.cpp 3KB
flutter_window.cpp 2KB
flutter_window.cpp 2KB
flutter_window.cpp 2KB
utils.cpp 2KB
utils.cpp 2KB
flutter_window.cpp 2KB
utils.cpp 2KB
utils.cpp 2KB
utils.cpp 2KB
utils.cpp 2KB
utils.cpp 2KB
main.cpp 1KB
main.cpp 1KB
main.cpp 1KB
main.cpp 1KB
main.cpp 1KB
main.cpp 1KB
main.cpp 1KB
language_subtag_registry.dart 690KB
test.dart 97KB
analyze.dart 95KB
perf_tests.dart 74KB
analyze_snippet_code.dart 46KB
create_api_docs.dart 45KB
prepare_package_test.dart 39KB
devices.dart 39KB
utils.dart 29KB
service_worker_test.dart 29KB
gen_localizations.dart 29KB
logical_key_data.dart 25KB
check_code_samples.dart 25KB
main_test.dart 24KB
module_test_ios.dart 24KB
build_ios_framework_module_test.dart 24KB
browser.dart 23KB
sector_layout.dart 22KB
vitool_test.dart 22KB
archive_creator.dart 20KB
update_icons.dart 19KB
unpublish_package.dart 19KB
vitool.dart 18KB
plugin_tests.dart 18KB
analyze_test.dart 17KB
create_api_docs_test.dart 17KB
localizations_utils.dart 16KB
plugin_lint_mac.dart 15KB
apk_utils.dart 15KB
keyboard_maps_code_gen.dart 15KB
dynamic_content_color.0.dart 15KB
module_test.dart 15KB
gallery.dart 15KB
main.dart 14KB
cocoon_test.dart 14KB
physical_key_data.dart 14KB
parent_data.0.dart 13KB
star_border.0.dart 12KB
run.dart 12KB
hot_mode_tests.dart 12KB
gen_defaults_test.dart 12KB
time_picker_template.dart 12KB
navigation_bar.2.dart 12KB
codesign_test.dart 12KB
input_decorator_template.dart 12KB
gen_keycodes.dart 11KB
icon_button_template.dart 11KB
linear_border.0.dart 11KB
check_code_samples_test.dart 11KB
gradle_plugin_light_apk_test.dart 11KB
run_tests.dart 11KB
show_time_picker.0.dart 11KB
module_custom_host_app_name_test.dart 11KB
selectable_region.0.dart 11KB
runner.dart 11KB
ab.dart 11KB
framework.dart 11KB
template.dart 10KB
date_picker_template.dart 10KB
input_chip.1.dart 10KB
gen_defaults.dart 10KB
plugin_dependencies_test.dart 10KB
web_benchmarks.dart 9KB
motion_events_page.dart 9KB
text_editing_integration.dart 9KB
slotted_multi_child_render_object_widget_mixin.0.dart 9KB
utils.dart 9KB
gen_date_localizations.dart 9KB
cocoon.dart 9KB
isolate.dart 9KB
utils.dart 9KB
menu_template.dart 9KB
共 2000 条
- 1
- 2
- 3
- 4
- 5
- 6
- 20
资源评论
芝麻粒儿
- 粉丝: 6w+
- 资源: 2万+
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功