Skip to content

D
dlink
Commit 5d7386d5 authored by walkhan's avatar walkhan
Browse files
Options

add integrate and openapi document

parent be52b5e8
Hide whitespace changes
Inline Side-by-side
Showing with 448 additions and 19 deletions
docs/zh-CN/_sidebar.md
View file @ 5d7386d5
...@@ -10,7 +10,7 @@ ...@@ -10,7 +10,7 @@
- [编译](/zh-CN/quick_start/build.md) - [编译](/zh-CN/quick_start/build.md)
- [部署](/zh-CN/quick_start/deploy.md) - [部署](/zh-CN/quick_start/deploy.md)
- 使用手册 - 使用手册
- FlinkSQL Studio - 数据开发
- [Studio介绍](/zh-CN/administrator-guide/Studio/studio_introduce.md) - [Studio介绍](/zh-CN/administrator-guide/Studio/studio_introduce.md)
- 作业管理 - 作业管理
- 作业开发 - 作业开发
...@@ -30,28 +30,29 @@ ...@@ -30,28 +30,29 @@
- 运维中心 - 运维中心
- 生命周期管理 - 生命周期管理
- 作业监控 - 作业监控
- openapi - [OpenAPI](/zh-CN/api/openapi.md)
- 扩展 - 扩展
- [集成](/zh-CN/extend/integrate.md)
- 功能扩展 - 功能扩展
- [扩展Flink版本](/zh-CN/extend/flinkversion.md) - [扩展Flink版本](/zh-CN/extend/flinkversion.md)
- [扩展连接器](/zh-CN/extend/connector.md) - [扩展连接器](/zh-CN/extend/connector.md)
- [扩展数据源](/zh-CN/extend/datasource.md) - [扩展数据源](/zh-CN/extend/datasource.md)
- [FlinkSQL 编辑器自动补全函数](/zh-CN/extend/completion.md) - [FlinkSQL 编辑器自动补全函数](/zh-CN/extend/completion.md)
- 集成生态 - 实践指南
- [Flink-CDC集成](/zh-CN/extend/flinkcdc.md) - [Yarn提交实践指南](/zh-CN/practice/yarnsubmit.md)
- [Flink-CDC-Kafka多源合并](/zh-CN/extend/Flink_CDC_kafka_Multi_source_merger.md) - [Dlink 核心概念及实现原理详解](/zh-CN/practice/principle.md)
- [hive集成](/zh-CN/extend/hive.md) - [AGGTABLE 表值聚合的实践](/zh-CN/practice/aggtable.md)
- [clickhouse集成](/zh-CN/extend/clickhouse.md) - 集成指南
- [Doris集成](/zh-CN/extend/doris.md) - [Flink-CDC集成](/zh-CN/extend/flinkcdc.md)
- [Hudi集成](/zh-CN/extend/hudi.md) - [Flink-CDC-Kafka多源合并](/zh-CN/extend/Flink_CDC_kafka_Multi_source_merger.md)
- [Iceberg集成](/zh-CN/extend/iceberg.md) - [hive集成](/zh-CN/extend/hive.md)
- [Flink UDF集成](/zh-CN/extend/udf.md) - [clickhouse集成](/zh-CN/extend/clickhouse.md)
- [DolphinScheduler集成](/zh-CN/extend/dolphinscheduler.md) - [Doris集成](/zh-CN/extend/doris.md)
- [DataSphereStudio集成](/zh-CN/extend/dataspherestudio.md) - [Hudi集成](/zh-CN/extend/hudi.md)
- 实践分享 - [Iceberg集成](/zh-CN/extend/iceberg.md)
- [Yarn提交实践指南](/zh-CN/practice/yarnsubmit.md) - [Flink UDF集成](/zh-CN/extend/udf.md)
- [Dlink 核心概念及实现原理详解](/zh-CN/practice/principle.md) - [DolphinScheduler集成](/zh-CN/extend/dolphinscheduler.md)
- [AGGTABLE 表值聚合的实践](/zh-CN/practice/aggtable.md) - [DataSphereStudio集成](/zh-CN/extend/dataspherestudio.md)
- 开发者指南 - 开发者指南
- 参与贡献 - 参与贡献
- [如何贡献](/zh-CN/developer-guide/how_contribute.md) - [如何贡献](/zh-CN/developer-guide/how_contribute.md)
......
docs/zh-CN/api/openapi.md
View file @ 5d7386d5
## 敬请期待 Dinky平台提供OpenAPI能力，通过调用Dinky的OpenAPI可以使用Dinky的功能，实现应用和Dinky的集成和交互。
\ No newline at end of file
### 背景
Dinky的OpenAPI提供了10多种API功能。通过调用API，可以快速进行操作和系统集成对接等工作，提高数据开发效率，满足企业定制化需求。您还可以通过开放平台，轻松获取OpenAPI的使用情况。
当前支持的OpenAPI包括如下：
| 序号 | 执行模式 | 类型名称 | 作用 |
| :--: | :------------: | :----------------: | :------------------------------: |
| 1 | explainSql | cancel | 调用取消FlinkSQL作业 |
| 2 | explainSql | statement | 调用statement语句 |
| 3 | executeJar | yarn-application | 调用执行yarn-application jar包 |
| 4 | executeSql | kubernetes-session | 调用执行作业在k8ssession上运行 |
| 5 | executeSql | local | 调用执行作业在local上运行 |
| 6 | executeSql | yarn-per-job | 调用执行作业在yarn-per-job上运行 |
| 7 | executeSql | standalone | 调用执行作业在standalone上运行 |
| 8 | executeSql | yarn-session | 调用执行作业在yarn-session上运行 |
| 9 | getJobData | | |
| 10 | getJobPlan | statement | 调用获取执行计划 |
| 11 | getStreamGraph | statement | 调用获取Flink DAG图 |
| 12 | savepoint | | 调用手动保存checkpoints |
| 13 | savepointTask | | 调用并触发savepoint |
| 14 | submitTask | | 作业调度 |
### 示例
**explainSql**
explainSql包括FlinkSQL作业取消和statement语句执行
FlinkSQL作业cancel：
**explainSql**
explainSql包括Fcancel
```
http://127.0.0.1:8888/openapi/explainSql
{
/* required-start */
"jobId":"195352b0a4518e16699983a13205f059",
/* required-end */
/* custom-start */
"address":"127.0.0.1:8081",
"gatewayConfig":{
"clusterConfig":{
"appId":"application_1637739262398_0032",
"flinkConfigPath":"/opt/src/flink-1.13.3_conf/conf",
"flinkLibPath":"hdfs:///flink13/lib/flinklib",
"yarnConfigPath":"/usr/local/hadoop/hadoop-2.7.7/etc/hadoop"
}
}
/* custom-start */
}
```
statement语句执行：
```
http://127.0.0.1:8888/openapi/explainSql
{
/* required-start */
"statement":"CREATE TABLE Orders (\r\n order_number INT,\r\n price DECIMAL(32,2),\r\n order_time TIMESTAMP(3)\r\n) WITH (\r\n 'connector' = 'datagen',\r\n 'rows-per-second' = '1',\r\n 'fields.order_number.kind' = 'sequence',\r\n 'fields.order_number.start' = '1',\r\n 'fields.order_number.end' = '1000'\r\n);\r\nCREATE TABLE pt (\r\nordertotal INT,\r\nnumtotal INT\r\n) WITH (\r\n 'connector' = 'print'\r\n);\r\ninsert into pt select 1 as ordertotal ,sum(order_number)*2 as numtotal from Orders",
/* required-end */
/* default-start */
"useStatementSet":false,
"fragment":false,
"parallelism":1,
/* default-start */
/* custom-start */
"configuration":{
"table.exec.resource.default-parallelism":2
}
/* custom-end */
}
```
**executeJar**
yarn-application jar包调用并提交
```
http://127.0.0.1:8888/openapi/executeJar
{
/* required-start */
"type":"yarn-application",
"gatewayConfig":{
"clusterConfig":{
"flinkConfigPath":"/opt/src/flink-1.13.3_conf/conf",
"flinkLibPath":"hdfs:///flink13/lib/flinklib",
"yarnConfigPath":"/usr/local/hadoop/hadoop-2.7.7/etc/hadoop"
},
"appConfig":{
"userJarPath":"hdfs:///flink12/jar/currencyAppJar.jar",
"userJarParas":["--id","2774,2775,2776"," --type","dwd"],
"userJarMainAppClass":"com.app.MainApp"
},
"flinkConfig": {
"configuration":{
"parallelism.default": 1
}
}
},
/* required-end */
/* custom-start */
"jobName":"openapitest",
"savePointPath":"hdfs://ns/flink/savepoints/savepoint-5f4b8c-4326844a6843"
/* custom-end */
}
```
**executeSql**
executeSql提交执行作业包括local，kubernetes-session，yarn-per-job，standalone，yarn-session
local:
```
http://127.0.0.1:8888/openapi/executeSql
{
/* required-start */
"type":"local",
"statement":"CREATE TABLE Orders (\r\n order_number INT,\r\n price DECIMAL(32,2),\r\n order_time TIMESTAMP(3)\r\n) WITH (\r\n 'connector' = 'datagen',\r\n 'rows-per-second' = '1',\r\n 'fields.order_number.kind' = 'sequence',\r\n 'fields.order_number.start' = '1',\r\n 'fields.order_number.end' = '1000'\r\n);\r\nCREATE TABLE pt (\r\nordertotal INT,\r\nnumtotal INT\r\n) WITH (\r\n 'connector' = 'print'\r\n);\r\ninsert into pt select 1 as ordertotal ,sum(order_number)*2 as numtotal from Orders",
/* required-end */
/* default-start */
"useResult":false,
"useChangeLog":false,
"useAutoCancel":false,
"useStatementSet":false,
"fragment":false,
"maxRowNum":100,
"checkPoint":0,
"parallelism":1,
/* default-start */
/* custom-start */
"jobName":"openapitest",
"savePointPath":"hdfs://ns/flink/savepoints/savepoint-5f4b8c-4326844a6843",
"configuration":{
"table.exec.resource.default-parallelism":2
}
/* custom-end */
}
```
kubernetes-session：
```
http://127.0.0.1:8888/openapi/executeSql
{
/* required-start */
"type":"kubernetes-session",
"address":"127.0.0.1:8081",
"statement":"CREATE TABLE Orders (\r\n order_number INT,\r\n price DECIMAL(32,2),\r\n order_time TIMESTAMP(3)\r\n) WITH (\r\n 'connector' = 'datagen',\r\n 'rows-per-second' = '1',\r\n 'fields.order_number.kind' = 'sequence',\r\n 'fields.order_number.start' = '1',\r\n 'fields.order_number.end' = '1000'\r\n);\r\nCREATE TABLE pt (\r\nordertotal INT,\r\nnumtotal INT\r\n) WITH (\r\n 'connector' = 'print'\r\n);\r\ninsert into pt select 1 as ordertotal ,sum(order_number)*2 as numtotal from Orders",
/* required-end */
/* default-start */
"useResult":false,
"useStatementSet":false,
"useChangeLog":false,
"useAutoCancel":false,
"fragment":false,
"maxRowNum":100,
"checkPoint":0,
"parallelism":1,
/* default-start */
/* custom-start */
"jobName":"openapitest",
"savePointPath":"hdfs://ns/flink/savepoints/savepoint-5f4b8c-4326844a6843",
"configuration":{
"table.exec.resource.default-parallelism":2
}
/* custom-end */
}
```
yarn-per-job:
```
http://127.0.0.1:8888/openapi/executeSql
{
/* required-start */
"type":"yarn-per-job",
"statement":"CREATE TABLE Orders (\r\n order_number INT,\r\n price DECIMAL(32,2),\r\n order_time TIMESTAMP(3)\r\n) WITH (\r\n 'connector' = 'datagen',\r\n 'rows-per-second' = '1',\r\n 'fields.order_number.kind' = 'sequence',\r\n 'fields.order_number.start' = '1',\r\n 'fields.order_number.end' = '1000'\r\n);\r\nCREATE TABLE pt (\r\nordertotal INT,\r\nnumtotal INT\r\n) WITH (\r\n 'connector' = 'print'\r\n);\r\ninsert into pt select 1 as ordertotal ,sum(order_number)*2 as numtotal from Orders",
"gatewayConfig":{
"clusterConfig":{
"flinkConfigPath":"/opt/src/flink-1.13.3_conf/conf",
"flinkLibPath":"hdfs:///flink13/lib/flinklib",
"yarnConfigPath":"/usr/local/hadoop/hadoop-2.7.7/etc/hadoop"
},
"flinkConfig": {
"configuration":{
"parallelism.default": 1
}
}
},
/* required-end */
/* default-start */
"useResult":false,
"useStatementSet":false,
"fragment":false,
"maxRowNum":100,
"checkPoint":0,
"parallelism":1,
/* default-start */
/* custom-start */
"jobName":"openapitest",
"savePointPath":"hdfs://ns/flink/savepoints/savepoint-5f4b8c-4326844a6843",
"configuration":{
"table.exec.resource.default-parallelism":2
}
/* custom-end */
}
```
yarn-session:
```
http://127.0.0.1:8888/openapi/executeSql
{
/* required-start */
"type":"yarn-session",
"address":"127.0.0.1:8081",
"statement":"CREATE TABLE Orders (\r\n order_number INT,\r\n price DECIMAL(32,2),\r\n order_time TIMESTAMP(3)\r\n) WITH (\r\n 'connector' = 'datagen',\r\n 'rows-per-second' = '1',\r\n 'fields.order_number.kind' = 'sequence',\r\n 'fields.order_number.start' = '1',\r\n 'fields.order_number.end' = '1000'\r\n);\r\nCREATE TABLE pt (\r\nordertotal INT,\r\nnumtotal INT\r\n) WITH (\r\n 'connector' = 'print'\r\n);\r\ninsert into pt select 1 as ordertotal ,sum(order_number)*2 as numtotal from Orders",
/* required-end */
/* default-start */
"useResult":false,
"useStatementSet":false,
"useChangeLog":false,
"useAutoCancel":false,
"fragment":false,
"maxRowNum":100,
"checkPoint":0,
"parallelism":1,
/* default-start */
/* custom-start */
"jobName":"openapitest",
"savePointPath":"hdfs://ns/flink/savepoints/savepoint-5f4b8c-4326844a6843",
"configuration":{
"table.exec.resource.default-parallelism":2
}
/* custom-end */
}
```
standalone:
```
http://127.0.0.1:8888/openapi/executeSql
{
/* required-start */
"type":"standalone",
"address":"127.0.0.1:8081",
"statement":"CREATE TABLE Orders (\r\n order_number INT,\r\n price DECIMAL(32,2),\r\n order_time TIMESTAMP(3)\r\n) WITH (\r\n 'connector' = 'datagen',\r\n 'rows-per-second' = '1',\r\n 'fields.order_number.kind' = 'sequence',\r\n 'fields.order_number.start' = '1',\r\n 'fields.order_number.end' = '1000'\r\n);\r\nCREATE TABLE pt (\r\nordertotal INT,\r\nnumtotal INT\r\n) WITH (\r\n 'connector' = 'print'\r\n);\r\ninsert into pt select 1 as ordertotal ,sum(order_number)*2 as numtotal from Orders",
/* required-end */
/* default-start */
"useResult":false,
"useStatementSet":false,
"useChangeLog":false,
"useAutoCancel":false,
"fragment":false,
"maxRowNum":100,
"checkPoint":0,
"parallelism":1,
/* default-start */
/* custom-start */
"jobName":"openapitest",
"savePointPath":"hdfs://ns/flink/savepoints/savepoint-5f4b8c-4326844a6843",
"configuration":{
"table.exec.resource.default-parallelism":2
}
/* custom-end */
}
```
**getJobData**
```
http://127.0.0.1:8888/openapi/getJobData?jobId=195352b0a4518e16699983a13205f059
```
**getJobPlan**
```
http://127.0.0.1:8888/openapi/getJobPlan
{
/* required-start */
"statement":"CREATE TABLE Orders (\r\n order_number INT,\r\n price DECIMAL(32,2),\r\n order_time TIMESTAMP(3)\r\n) WITH (\r\n 'connector' = 'datagen',\r\n 'rows-per-second' = '1',\r\n 'fields.order_number.kind' = 'sequence',\r\n 'fields.order_number.start' = '1',\r\n 'fields.order_number.end' = '1000'\r\n);\r\nCREATE TABLE pt (\r\nordertotal INT,\r\nnumtotal INT\r\n) WITH (\r\n 'connector' = 'print'\r\n);\r\ninsert into pt select 1 as ordertotal ,sum(order_number)*2 as numtotal from Orders",
/* required-end */
/* default-start */
"useStatementSet":false,
"fragment":false,
"parallelism":1,
/* default-start */
/* custom-start */
"configuration":{
"table.exec.resource.default-parallelism":2
}
/* custom-end */
}
```
**getStreamGraph**
```
http://127.0.0.1:8888/openapi/getStreamGraph
{
/* required-start */
"statement":"CREATE TABLE Orders (\r\n order_number INT,\r\n price DECIMAL(32,2),\r\n order_time TIMESTAMP(3)\r\n) WITH (\r\n 'connector' = 'datagen',\r\n 'rows-per-second' = '1',\r\n 'fields.order_number.kind' = 'sequence',\r\n 'fields.order_number.start' = '1',\r\n 'fields.order_number.end' = '1000'\r\n);\r\nCREATE TABLE pt (\r\nordertotal INT,\r\nnumtotal INT\r\n) WITH (\r\n 'connector' = 'print'\r\n);\r\ninsert into pt select 1 as ordertotal ,sum(order_number)*2 as numtotal from Orders",
/* required-end */
/* default-start */
"useStatementSet":false,
"fragment":false,
"parallelism":1,
/* default-start */
/* custom-start */
"configuration":{
"table.exec.resource.default-parallelism":2
}
/* custom-end */
}
```
**savepoint**
```
http://127.0.0.1:8888/openapi/savepoint
{
/* required-start */
"jobId":"195352b0a4518e16699983a13205f059",
"savePointType":"trigger", // trigger | stop | cancel
/* required-end */
/* custom-start */
"savePoint":"195352b0a4518e16699983a13205f059",
"address":"127.0.0.1:8081",
"gatewayConfig":{
"clusterConfig":{
"appId":"application_1637739262398_0032",
"flinkConfigPath":"/opt/src/flink-1.13.3_conf/conf",
"flinkLibPath":"hdfs:///flink13/lib/flinklib",
"yarnConfigPath":"/usr/local/hadoop/hadoop-2.7.7/etc/hadoop"
}
}
/* custom-start */
}
```
**savepointTask**
```
http://127.0.0.1:8888/openapi/savepointTask
{
/* required-start */
"taskId":"1",
/* required-end */
/* custom-start */
"type":"trigger" // trigger | stop | cancel | canceljob
/* custom-start */
}
```
**submitTask**
```
http://127.0.0.1:8888/openapi/submitTask?id=1
```
OpenAPI包括元数据、数据开发、数据集成、运维中心、数据质量、数据服务等。其中数据开发已经开发完成，其他根据版本发布会逐步实现。如果您在数据开发过程中需要使用Dinky OpenAPI，使用方式请参见示例。
\ No newline at end of file
docs/zh-CN/extend/integrate.md 0 → 100644
View file @ 5d7386d5
**声明:** Dinky并没有维护下面列出的库,其中有一部分是生产实践,一部分还未做过相关生产实践。欢迎使用Dinky已经集成的第三方库。
## 关系型数据库
- MySQL
- PostGreSQL
- Oracle
- SQLServer
## OLAP引擎
- Doris
- ClickHouse
- Phoenix
- Hive
## 消息队列
- Kafka
- Pulsar
以上均采用FlinkSQL Connector与Dinky集成使用，下面链接提供了FlinkSQL相关Connector包，请在[maven仓库](https://repo.maven.apache.org/maven2/org/apache/flink/)自行下载。
## CDC工具
- [Flink CDC](https://github.com/ververica/flink-cdc-connectors/releases/)
## 调度平台
- **Oozie**
- **dolphinscheduler:** [Apache dolphinscheduler](https://dolphinscheduler.apache.org/zh-cn/)大数据调度平台
对于调度平台,需支持HTTP服务即可
## 数据湖
- Hudi
- Iceberg
[Hudi](https://github.com/apache/hudi)及[Iceberg](https://github.com/apache/iceberg)请移步到github自行编译或者下载
## HADOOP
- [flink-shaded-hadoop-3-uber](https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/flink/flink-shaded-hadoop-3-uber)
## 其他
- **dataspherestudio:** 微众银行开源[数据中台](https://github.com/WeBankFinTech/DataSphereStudio)
与Dinky的集成实践请移步最佳实践中的集成指南
\ No newline at end of file
docs/zh-CN/quick_start/build.md
View file @ 5d7386d5
...@@ -25,7 +25,7 @@ npm install -g npm@7.19.0 ...@@ -25,7 +25,7 @@ npm install -g npm@7.19.0
### Linux 版本 ### Linux 版本
[下载地址]( <https://registry.npmmirror.com/-/binary/node/v14.19.0/node-v14.19.0.tar.gz>) [下载地址](https://nodejs.org/download/release/v14.19.1/node-v14.19.1-linux-x64.tar.gz)
下载完成后，安装配置即可，操作如下： 下载完成后，安装配置即可，操作如下：
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment