Presto大数据查询引擎是用于大数据的分布式SQL查询引擎

# Presto product tests The product tests make use of user visible interfaces (e.g. `presto-cli`) to test Presto for correctness. The product tests complement the unit tests because unlike the unit tests, they exercise the Presto codebase end-to-end. To keep the execution of the product tests as lightweight as possible we decided to use [Docker](http://www.docker.com/). The general execution setup is as follows: a single Docker container runs Hadoop in pseudo-distributed mode and Presto runs either in Docker container(s) (both pseudo-distributed and distributed setups are possible) or manually from IntelliJ (for debugging Presto). The tests run in a separate JVM and they can be started using the scripts found in `presto-product-tests/bin`. The product tests are run using the [Tempto](https://github.com/prestodb/tempto) harness. Developers should consider writing product tests in addition to any unit tests when making changes to user visible features. The product tests should also be run after all code changes to ensure no existing functionality has been broken. ## Requirements *Running the Presto product tests requires at least 4GB of free memory* ### GNU/Linux * [`docker >= 1.10`](https://docs.docker.com/installation/#installation) ``` wget -qO- https://get.docker.com/ | sh ``` * [`docker-compose >= 1.8.1`](https://docs.docker.com/compose/install/) ``` pip install docker-compose ``` ### OS X using Docker for Mac * Install [Docker for Mac](https://docs.docker.com/docker-for-mac/) * Add entries in `/etc/hosts` for all services running in docker containers: `hadoop-master`, `mysql`, `postgres`, `cassandra`, `presto-master`. They should point to your external IP address (shown by `ifconfig` on your Mac, not inside Docker). * The default memory setting of 2GB might not be sufficient for some profiles like `singlenode-ldap`. You may need 4-8 GB or even more to run certain tests. You can increase Docker memory by going to *Docker Preferences -> Advanced -> Memory*. ## Use the `docker-compose` wrappers We're using [multiple compose files](https://docs.docker.com/compose/extends/#multiple-compose-files) because of the number of overrides needed for different environments, and deficiencies of `extends:` syntax (see the note [here](https://docs.docker.com/compose/extends/#extending-services)). To ease the pain of passing multiple `-f` arguments to `docker-compose`, each environment has a `compose.sh` wrapper script. Thanks to it, instead of e.g. `docker-compose -f ./docker-compose.yml -f ../common/standard.yml [compose commands]` one can simply write `compose.sh [compose commands]` ## Running the product tests The Presto product tests must be run explicitly because they do not run as part of the Maven build like the unit tests do. Note that the product tests cannot be run in parallel. This means that only one instance of a test can be run at once in a given environment. To run all product tests and exclude the `quarantine`, `big_query` and `profile_specific_tests` groups run the following command: ``` ./mvnw install -DskipTests presto-product-tests/bin/run_on_docker.sh <profile> -x quarantine,big_query,profile_specific_tests ``` where profile is one of either: #### Profiles - **multinode** - pseudo-distributed Hadoop installation running on a single Docker container and a distributed Presto installation running on multiple Docker containers. For multinode the default configuration is 1 coordinator and 1 worker. - **multinode-tls** - pseudo-distributed Hadoop installation running on a single Docker container and a distributed Presto installation running on multiple Docker containers. Presto is configured to only accept connections on the HTTPS port (7878), and both coordinator and worker traffic is encrypted. For multinode-tls, the default configuration is 1 coordinator and 2 workers. - **multinode-tls-kerberos** - pseudo-distributed Hadoop installation running on a single Docker container and a distributed installation of kerberized Presto running on multiple Docker containers. Presto is configured to only accept connections on the HTTPS port (7778), and both coordinator and worker traffic is encrypted and kerberized. For multinode-tls-kerberos, the default configuration is 1 coordinator and 2 workers. - **singlenode** - pseudo-distributed Hadoop installation running on a single Docker container and a single node installation of Presto also running on a single Docker container. - **singlenode-hdfs-impersonation** - HDFS impersonation enabled on top of the environment in singlenode profile. Presto impersonates the user who is running the query when accessing HDFS. - **singlenode-kerberos-hdfs-impersonation** - pseudo-distributed kerberized Hadoop installation running on a single Docker container and a single node installation of kerberized Presto also running on a single Docker container. This profile has Kerberos impersonation. Presto impersonates the user who is running the query when accessing HDFS. - **singlenode-kerberos-hdfs-no-impersonation** - pseudo-distributed Hadoop installation running on a single Docker container and a single node installation of kerberized Presto also running on a single Docker container. This profile runs Kerberos without impersonation. - **singlenode-ldap** - Three single node Docker containers, one running an OpenLDAP server, one running with SSL/TLS certificates installed on top of a single node Presto installation, and one with a pseudo-distributed Hadoop installation. - **singlenode-sqlserver** - pseudo-distributed Hadoop installation running on a single Docker container, a single node installation of Presto also running on a single Docker container and one running SQL Server server. While running tests on ``singlenode-sqlserver`` make sure to exclude `mysql_connector` and `postgresql_connector` tests i.e. `-x mysql_connector, postgresql_connector`. ### Hadoop docker image used for testing The default Hadoop/Hive docker image used for testing is defined in `conf/common/compose-commons.sh` and can be controlled via the `HADOOP_BASE_IMAGE` and `DOCKER_IMAGES_VERSION` env variables. - `HADOOP_BASE_IMAGE` defines the Hadoop distribution family (as found in [PrestoDB Hadoop docker repo](https://cloud.docker.com/swarm/prestodb/repository/list?name=hive&namespace=prestodb)). The name should be without the `-kerberized` suffix, eg. `cdh5.13-hive`. Only images that have their kerberized counterparts can be used with test profiles implying a kerberized environment eg. `singlenode-kerberos-hdfs-impersonation`, you should still use the base name for this env variable, the `-kerberized` suffix will be added automatically. - `DOCKER_IMAGES_VERSION` determines the version of the images used, both Hadoop images and base Centos images to host Presto, and serve as various run environments throughout the tests. Versions can be found on the [PrestoDB docker repo](https://cloud.docker.com/swarm/prestodb/repository/list) as well. You may use any version, either release or snapshot. Note that all images will be required to have this version, because this version is used globally. This is to ease maintenance and simplify debugging. Please keep in mind that if you run tests on Hive of version not greater than 1.0.1, you should exclude test from `post_hive_1_0_1` group by passing the following flag to tempto: `-x post_hive_1_0_1`. First version of Hive capable of running tests from `post_hive_1_0_1` group is Hive 1.1.0. For more information on the various ways in which Presto can be configured to interact with Kerberized Hive and Hadoop, please refer to the [Hive connector documentation](https://prestodb.github.io/docs/current/connector/hive.html). ### Running a single test The `run_on_docker.sh` script can also run individual product tests. Presto product tests are either [Java based](https://github.com/prestodb/tempto#java-based-tests) or [convention based](ht