<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-->
# Apache Kyuubi (Incubating)
<img src="https://svn.apache.org/repos/asf/comdev/project-logos/originals/kyuubi-1.svg" alt="Kyuubi logo" height="120px" align="right" />
[![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)
[![Release](https://img.shields.io/github/v/release/apache/incubator-kyuubi?label=release)](https://github.com/apache/incubator-kyuubi/releases)
[![](https://tokei.rs/b1/github.com/apache/incubator-kyuubi)](https://github.com/apache/incubator-kyuubi)
[![codecov](https://codecov.io/gh/apache/incubator-kyuubi/branch/master/graph/badge.svg)](https://codecov.io/gh/apache/incubator-kyuubi)
![GitHub Workflow Status](https://img.shields.io/github/workflow/status/apache/incubator-kyuubi/Kyuubi/master?style=plastic)
[![Travis](https://api.travis-ci.com/apache/incubator-kyuubi.svg?branch=master)](https://travis-ci.com/apache/incubator-kyuubi)
[![Documentation Status](https://readthedocs.org/projects/kyuubi/badge/?version=latest)](https://kyuubi.apache.org/docs/latest/)
![GitHub top language](https://img.shields.io/github/languages/top/apache/incubator-kyuubi)
[![Commit activity](https://img.shields.io/github/commit-activity/m/apache/incubator-kyuubi)](https://github.com/apache/incubator-kyuubi/graphs/commit-activity)
[![Average time to resolve an issue](http://isitmaintained.com/badge/resolution/apache/incubator-kyuubi.svg)](http://isitmaintained.com/project/apache/incubator-kyuubi "Average time to resolve an issue")
[![Percentage of issues still open](http://isitmaintained.com/badge/open/apache/incubator-kyuubi.svg)](http://isitmaintained.com/project/apache/incubator-kyuubi "Percentage of issues still open")
## What is Kyuubi?
Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for large-scale data management, processing, and analytics, built on top of Apache Spark and designed to support more engines (i.e., Flink). It has been open-sourced by NetEase since 2018. We are aiming to make Kyuubi an "out-of-the-box" tool for data warehouses and data lakes.
Kyuubi provides a pure SQL gateway through Thrift JDBC/ODBC interface for end-users to manipulate large-scale data with pre-programmed and extensible Spark SQL engines. This "out-of-the-box" model minimizes the barriers and costs for end-users to use Spark at the client side. At the server-side, Kyuubi server and engines' multi-tenant architecture provides the administrators a way to achieve computing resource isolation, data security, high availability, high client concurrency, etc.
![](./docs/imgs/kyuubi_positioning.png)
- [x] A HiveServer2-like API
- [x] Multi-tenant Spark Support
- [x] Running Spark in a serverless way
### Target Users
Kyuubi's goal is to make it easy and efficient for `anyone` to use Spark(maybe other engines soon) and facilitate users to handle big data like ordinary data. Here, `anyone` means that users do not need to have a Spark technical background but a human language, SQL only. Sometimes, SQL skills are unnecessary when integrating Kyuubi with Apache Superset, which supports rich visualizations and dashboards.
In typical big data production environments with Kyuubi, there should be system administrators and end-users.
- System administrators: A small group consists of Spark experts responsible for Kyuubi deployment, configuration, and tuning.
- End-users: Focus on business data of their own, not where it stores, how it computes.
Additionally, the Kyuubi community will continuously optimize the whole system with various features, such as History-Based Optimizer, Auto-tuning, Materialized View, SQL Dialects, Functions, e.t.c.
### Usage scenarios
#### Port workloads from HiveServer2 to Spark SQL
In typical big data production environments, especially secured ones, all bundled services manage access control lists to restricting access to authorized users. For example, Hadoop YARN divides compute resources into queues. With Queue ACLs, it can identify and control which users/groups can take actions on particular queues. Similarly, HDFS ACLs control access of HDFS files by providing a way to set different permissions for specific users/groups.
Apache Spark is a unified analytics engine for large-scale data processing. It provides a Distributed SQL Engine, a.k.a, the Spark Thrift Server(STS), designed to be seamlessly compatible with HiveServer2 and get even better performance.
HiveServer2 can identify and authenticate a caller, and then if the caller also has permissions for the YARN queue and HDFS files, it succeeds. Otherwise, it fails. However, on the one hand, STS is a single Spark application. The user and queue to which STS belongs are uniquely determined at startup. Consequently, STS cannot leverage cluster managers such as YARN and Kubernetes for resource isolation and sharing or control the access for callers by the single user inside the whole system. On the other hand, the Thrift Server is coupled in the Spark driver's JVM process. This coupled architect puts a high risk on server stability and makes it unable to handle high client concurrency or apply high availability such as load balancing as it is stateful.
Kyuubi extends the use of STS in a multi-tenant model based on a unified interface and relies on the concept of multi-tenancy to interact with cluster managers to finally gain the ability of resources sharing/isolation and data security. The loosely coupled architecture of the Kyuubi server and engine dramatically improves the client concurrency and service stability of the service itself.
#### DataLake/LakeHouse Support
The vision of Kyuubi is to unify the portal and become an easy-to-use data lake management platform. Different kinds of workloads, such as ETL processing and BI analytics, can be supported by one platform, using one copy of data, with one SQL interface.
- Logical View support via Kyuubi DataLake Metadata APIs
- Multiple Catalogs support
- SQL Standard Authorization support for DataLake(coming)
#### Cloud Native Support
Kyuubi can deploy its engines on different kinds of Cluster Managers, such as, Hadoop YARN, Kubernetes, etc.
![](./docs/imgs/kyuubi_migrating_yarn_to_k8s.png)
### The Kyuubi Ecosystem(present and future)
The figure below shows our vision for the Kyuubi Ecosystem. Some of them have been realized, some in development, and others would not be possible without your help.
![](./docs/imgs/kyuubi_ecosystem.png)
## Online Documentation
Since Kyuubi 1.3.0-incubating, the Kyuubi online documentation is hosted by [https://kyuubi.apache.org/](https://kyuubi.apache.org/).
You can find the latest Kyuubi documentation on [this web page](https://kyuubi.apache.org/docs/latest/).
For 1.2 and earlier versions, please check the [Readthedocs](https://kyuubi.readthedocs.io/en/v1.2.0/) directly.
## Quick Start
Ready? [Getting Started](https://kyuubi.apache.org/docs/latest/quick_start/quick_start.html) with Kyuubi.
## [Contributing](./CONTRIBUTING.md)
## Contributor over time
[![Contributor over time](https://contributor-graph-api.apiseven.com/contributors-svg?chart=contributorOverTime&repo=apache/incubator-kyuubi)](https://api7.ai/contribut
没有合适的资源?快使用搜索试试~ 我知道了~
incubator-kyuubi-1.6.0-incubating.zip
共2000个文件
scala:772个
sql:367个
schema:260个
1.该资源内容由用户上传,如若侵权请联系客服进行举报
2.虚拟产品一经售出概不退款(资源遇到问题,请及时私信上传者)
2.虚拟产品一经售出概不退款(资源遇到问题,请及时私信上传者)
版权申诉
0 下载量 52 浏览量
2022-10-25
09:46:56
上传
评论
收藏 16.99MB ZIP 举报
温馨提示
Apache Kyuubi源码(incubator-kyuubi-1.6.0-incubating.zip)
资源推荐
资源详情
资源评论
收起资源包目录
incubator-kyuubi-1.6.0-incubating.zip (2000个子文件)
org.apache.spark.status.AppHistoryServerPlugin 45B
org.apache.kyuubi.engine.ApplicationOperation 940B
make.bat 2KB
beeline 2KB
kyuubi-defaults.conf 932B
.scalafmt.conf 715B
semantic.min.css 614KB
swagger-ui.css 140KB
icon.min.css 65KB
custom.css 1KB
dependencyList 10KB
DISCLAIMER 552B
dist 10KB
Dockerfile 3KB
Dockerfile 2KB
Dockerfile 1KB
.dockerignore 839B
kyuubi_layers.drawio 446KB
kyuubi_ecosystem.drawio 282KB
kyuubi_migrating_yarn_to_k8s.drawio 195KB
ha.drawio 101KB
engine_lifecycle.drawio 15KB
engine_share_level_group.drawio 13KB
engine_share_level_connection.drawio 13KB
engine_share_level_user.drawio 13KB
engine_share_level_server.drawio 12KB
localshufflereader.drawio 3KB
dra_task_fin.drawio 1KB
dra_executor_removal.drawio 1KB
dra_executor_add_ratio.drawio 1KB
dra_executor_added.drawio 1KB
dra_task_pending.drawio 1KB
java.sql.Driver 822B
icons.eot 59KB
KyuubiSparkSQL.g4 5KB
.gitattributes 1KB
.gitignore 2KB
org.apache.kyuubi.credentials.HadoopDelegationTokenProvider 982B
org.apache.kyuubi.credentials.HadoopDelegationTokenProvider 905B
q2.output.hash 816B
q98.output.hash 815B
q76.output.hash 815B
q27.output.hash 815B
q36.output.hash 815B
q57.output.hash 815B
q11.output.hash 815B
q96.output.hash 814B
q65.output.hash 814B
q59.output.hash 814B
q40.output.hash 814B
q94.output.hash 814B
q21.output.hash 814B
q38.output.hash 814B
q5.output.hash 814B
q16.output.hash 814B
q20.output.hash 814B
q77.output.hash 814B
q67.output.hash 814B
q79.output.hash 814B
q95.output.hash 814B
q10.output.hash 814B
q7.output.hash 814B
q3.output.hash 814B
q47.output.hash 814B
q74.output.hash 814B
q86.output.hash 814B
q46.output.hash 814B
q15.output.hash 814B
q12.output.hash 814B
q80.output.hash 814B
q34.output.hash 814B
q17.output.hash 814B
q60.output.hash 814B
q32.output.hash 814B
q72.output.hash 814B
q75.output.hash 814B
q30.output.hash 814B
q42.output.hash 814B
q14a.output.hash 814B
q43.output.hash 814B
q89.output.hash 814B
q19.output.hash 814B
q9.output.hash 814B
q1.output.hash 814B
q10.output.hash 814B
q7.output.hash 814B
q3.output.hash 814B
q13.output.hash 814B
q15.output.hash 814B
q6.output.hash 813B
q50.output.hash 813B
q78.output.hash 813B
q33.output.hash 813B
q28.output.hash 813B
q69.output.hash 813B
q19.output.hash 813B
q9.output.hash 813B
q18.output.hash 813B
q1.output.hash 813B
q71.output.hash 813B
共 2000 条
- 1
- 2
- 3
- 4
- 5
- 6
- 20
资源评论
YunFeiDong
- 粉丝: 33
- 资源: 3849
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功