# Apache Spark
Spark is a fast and general cluster computing system for Big Data. It provides
high-level APIs in Scala, Java, Python, and R, and an optimized engine that
supports general computation graphs for data analysis. It also supports a
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
MLlib for machine learning, GraphX for graph processing,
and Spark Streaming for stream processing.
<http://spark.apache.org/>
## Online Documentation
You can find the latest Spark documentation, including a programming
guide, on the [project web page](http://spark.apache.org/documentation.html).
This README file only contains basic setup instructions.
## Building Spark
Spark is built using [Apache Maven](http://maven.apache.org/).
To build Spark and its example programs, run:
build/mvn -DskipTests clean package
(You do not need to do this if you downloaded a pre-built package.)
More detailed documentation is available from the project site, at
["Building Spark"](http://spark.apache.org/docs/latest/building-spark.html).
For general development tips, including info on developing Spark using an IDE, see ["Useful Developer Tools"](http://spark.apache.org/developer-tools.html).
## Interactive Scala Shell
The easiest way to start using Spark is through the Scala shell:
./bin/spark-shell
Try the following command, which should return 1000:
scala> sc.parallelize(1 to 1000).count()
## Interactive Python Shell
Alternatively, if you prefer Python, you can use the Python shell:
./bin/pyspark
And run the following command, which should also return 1000:
>>> sc.parallelize(range(1000)).count()
## Example Programs
Spark also comes with several sample programs in the `examples` directory.
To run one of them, use `./bin/run-example <class> [params]`. For example:
./bin/run-example SparkPi
will run the Pi example locally.
You can set the MASTER environment variable when running examples to submit
examples to a cluster. This can be a mesos:// or spark:// URL,
"yarn" to run on YARN, and "local" to run
locally with one thread, or "local[N]" to run locally with N threads. You
can also use an abbreviated class name if the class is in the `examples`
package. For instance:
MASTER=spark://host:7077 ./bin/run-example SparkPi
Many of the example programs print usage help if no params are given.
## Running Tests
Testing first requires [building Spark](#building-spark). Once Spark is built, tests
can be run using:
./dev/run-tests
Please see the guidance on how to
[run tests for a module, or individual tests](http://spark.apache.org/developer-tools.html#individual-tests).
There is also a Kubernetes integration test, see resource-managers/kubernetes/integration-tests/README.md
## A Note About Hadoop Versions
Spark uses the Hadoop core library to talk to HDFS and other Hadoop-supported
storage systems. Because the protocols have changed in different versions of
Hadoop, you must build Spark against the same version that your cluster runs.
Please refer to the build documentation at
["Specifying the Hadoop Version and Enabling YARN"](http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn)
for detailed guidance on building for a particular distribution of Hadoop, including
building for particular Hive and Hive Thriftserver distributions.
## Configuration
Please refer to the [Configuration Guide](http://spark.apache.org/docs/latest/configuration.html)
in the online documentation for an overview on how to configure Spark.
## Contributing
Please review the [Contribution to Spark guide](http://spark.apache.org/contributing.html)
for information on how to get started contributing to the project.
没有合适的资源?快使用搜索试试~ 我知道了~
资源详情
资源评论
资源推荐
收起资源包目录
spark2.4.8_on_hadoop3.3.3 (1929个子文件)
.__common_metadata 176B
.__metadata 176B
.__static 176B
.__SUCCESS 176B
.__SUCCESS 176B
.__templates 176B
._als 176B
._apache 176B
._apache 176B
._b=0 176B
._b=1 176B
._beeline 176B
._bin 176B
._c=0 176B
._c=1 176B
._clickstream 176B
._cls=kittens 176B
._cls=multichannel 176B
_common_metadata 210B
._conf 176B
._data 176B
._date=2018-01 176B
._date=2018-01 176B
._date=2018-02 176B
._date=2018-02 176B
._day=1 176B
._day=1 176B
._day=25 176B
._day=26 176B
._docs 176B
._examples 176B
._examples 176B
._find-spark-home 176B
._graphx 176B
._graphx 176B
._hello 176B
._hive 176B
._hive 176B
._images 176B
._java 176B
._kittens 176B
._lib 176B
._LICENSE 176B
._licenses 176B
._linalg 176B
._linalg 176B
._main 176B
._Makefile 176B
_metadata 743B
._ml 176B
._ml 176B
._ml 176B
._ml 176B
._ml 176B
._mllib 176B
._mllib 176B
._mllib 176B
._mllib 176B
._mllib 176B
._month=10 176B
._month=9 176B
._month=9 176B
._multi-channel 176B
._NOTICE 176B
._orc_partitioned 176B
._org 176B
._org 176B
._origin 176B
._param 176B
._parquet_partitioned 176B
._partitioned 176B
._pylintrc 176B
._pyspark 176B
._pyspark 176B
._python 176B
._python 176B
._pythonconverters 176B
._r 176B
._resources 176B
._ridge-data 176B
._run-example 176B
._run-tests 176B
._run-tests-with-coverage 176B
._sbin 176B
._scala 176B
._spark 176B
._spark 176B
._spark-class 176B
._spark-shell 176B
._spark-sql 176B
._spark-submit 176B
._sparkR 176B
._sql 176B
._sql 176B
._sql 176B
._sql 176B
._sql 176B
._stat 176B
._streaming 176B
._streaming 176B
共 1929 条
- 1
- 2
- 3
- 4
- 5
- 6
- 20
姜上清风
- 粉丝: 18
- 资源: 3
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功
评论5