engine-main.zip

# Flutter Engine Build Definition Language The ***Flutter Engine Build Definition Language*** describes a build on CI by defining a combination of *sub-builds*, *archives*, *generators* and *dependencies*. It makes it simple to shard sub-builds by mapping build inputs to workflows, and listing the sub-build-generated artifacts explicitly. The Build Definition Language, Engine Recipes V2 and the generation of artifacts using GN+Ninja set the groundwork for efficient builds with dependency reusability. ## Glossary * **[recipes](https://github.com/luci/recipes-py)** - domain specific language for specifying sequences of subprocess calls in a cross-platform and testable way. * **Generator** - scripts in Dart, python or bash that combine the output of sub-builds to generate artifacts. * **Builder** - a combination of configuration, recipes and a given commit to build and test artifacts. * **Build** - a builder running with specific properties, repository and commit. * **[GN](https://gn.googlesource.com/gn/)** - a meta-build system that generates build files for [Ninja](https://ninja-build.org/). * **[Ninja](https://ninja-build.org)** - Ninja is a small build system with a focus on speed. * **CAS** - a service that stores arbitrary binary blobs addressed by (hash of) their content. It is specialized for low latency, high volume query/read/write operations. ## USAGE EXAMPLES Engine build definition files using the Build Definition Language can be found in the [flutter/engine/ci/builders](https://github.com/flutter/engine/tree/main/ci/builders) directory. The [engine orchestrator recipe](https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/engine_v2/) reads each file in that directory, shards their builds, collects artifacts and uploads them to the Google Cloud Storage bucket. The [.ci.yaml file](https://github.com/flutter/engine/blob/main/.ci.yaml) at the root of the `flutter/engine` repository puts all the components together. Builds are specified in that file using a property pointing to the build definition file to be used by engine\_v2 recipes. Full documentation of the `.ci.yaml` file format can be found [in the Cocoon repository here](https://github.com/flutter/cocoon/blob/main/CI_YAML.md). The following is a sample build configuration referencing [android\_aot\_engine.json](https://github.com/flutter/engine/blob/main/ci/builders/mac_android_aot_engine.json) in the `config_name` under `properties`: ```yaml - name: Mac mac_android_aot_engine recipe: engine_v2/engine_v2 timeout: 60 properties: config_name: mac_android_aot_engine $flutter/osx_sdk : >- { "sdk_version": "15a240d" } ``` ## Build Definition Language Assumptions To keep the build definition language simple the following assumptions were made during its design: * A build can be expressed as a set of independent sub-builds. * A sub-build can be defined as a sequence of a `gn` configuration step, a `ninja` build step, followed by self-contained test scripts, and self-contained generator scripts. All the elements are optional allowing to use gn+ninja without generators or generators without gn+ninja. * All the sub-builds required by a global generator are defined within the same configuration file. ## Build configuration file The build configuration is a json file containing a list of builds, tests, generators and archives. The following is an example of an empty configuration file: ```json { "builds": [], "tests": [], "generators": { "tasks": [] }, "archives": [ ] } ``` Note: tests, generators and archives can be omited if empty. Build configuration files have to be checked into the [engine_checkout/ci/builder](https://github.com/flutter/engine/tree/main/ci/builders) directory where engine v2 recipes will be reading them from. Configurations with a single build are supported. Single build configurations are located have to be checked into the [engine_checkout/ci/builder/standalone](https://github.com/flutter/engine/tree/main/ci/builders/standalone) A configuration file defines a top-level builder that will show up as a column in the [Flutter Dashboard](https://flutter-dashboard.appspot.com/#/build?repo=engine&branch=master). ### Magic variables Magic variables are special environment variables that can be used as parameters for generators and test commands in the local and global contexts. Magic environment variables have the following limitations: only `${FLUTTER_LOGS_DIR}` is currently supported and it needs to be used alone within the parameter string(e.g. `["${FLUTTER_LOGS_DIR}"]` is OK but `["path=${FLUTTER_LOGS_DIR}"]` is not). The current list of supported magic variables is: * `${FLUTTER_LOGS_DIR}` - translated to the path of the temporary folder where logs are being placed. * `${LUCI_WORKDIR}` - translated to the LUCI chroot working directory. * `${LUCI_CLEANUP}` - translated to the LUCI chroot temp directory. * `${REVISION}` - translated to the engine commit in postsubmit. In presubmit it is translated to an empty string. ### Build A build is a dictionary with a gn command, a ninja command, zero or more generator commands, zero or more local tests, zero or more local generators and zero or more output artifacts. The following is the high level structure of the build component: ```json { "archives": [], "drone_dimensions": [], "gclient_variables": {}, "gn": [], "name": "host_debug", "generators": [], "ninja": {}, "tests": [] "postsubmit_overrides": {} } ``` Each build element will be translated to an independent sub-build and its entire out directory will be uploaded to CAS. `gn`, `ninja`, `generators`, `tests` and `postsubmit_overrides` properties are optional. Gn and ninja properties can be used without generators or tests. Generators with no gn and ninja properties is also supported. #### Archives An archive component is used to tell the recipes which artifacts are generated by the build and where to upload them. By default the build output is archived to CAS in order to be used as a dependency for global tests. If no CAS archive is required `cas_archive": false,` needs to be added to the configuration. ```json { "name": "host_debug", "base_path": "out/host_debug/zip_archives/", "type": "gcs", "include_paths": [ "out/host_debug/zip_archives/linux-x64/artifacts.zip" ], "realm": "production" } ``` Description of the fields: * **name:** - by default the entire build output is uploaded to CAS. `name` is used to associate the CAS hash to a value that can be referenced later as a dependency of global tests. Name is also used to select the folder from within src/out to upload to CAS. e.g if the build generates src/out/host_debug name must be `host_debug`. * **base\_path:** - the portion of the path to remove from the full path before uploading to its final destination. In the example the above the base\_path **“out/host\_debug/zip\_archives”** will be removed from the include path **"out/host\_debug/zip\_archives/linux-x64/artifacts.zip"** before uploading to GCS, e.g. <bucket>/flutter/<commit>/linux-x64/artifacts.zip. * **Type:** - the type of storage to use. Currently only **“gcs”** and **“cas”** are supported. "gcs" uploads artifacts to GCS and "cas" to archive to CAS service. Cas value is used during development where we need to inspect the generated artifacts without worrying about location or cleanups. Gcs is expected for any artifacts being consumed by the flutter tool. * **Include\_paths:** - a list of strings with the paths to be uploaded to a given destination. * **cas_archive** - a boolean value indicating whether the build output will be archived to CAS or not. The default value is true. * **realm** - a string value of either `production` or `experimental` where production means the artifact will be uploaded to the