Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Contribute to GitLab
Sign in / Register
Toggle navigation
D
dlink
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
zhaowei
dlink
Commits
2a2d4f6b
Commit
2a2d4f6b
authored
Jan 13, 2022
by
wenmo
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
UDFJava 调试与加载
parent
f9cfbc03
Changes
20
Expand all
Show whitespace changes
Inline
Side-by-side
Showing
20 changed files
with
4317 additions
and
5 deletions
+4317
-5
README.md
README.md
+2
-2
TaskService.java
dlink-admin/src/main/java/com/dlink/service/TaskService.java
+2
-0
StudioServiceImpl.java
...c/main/java/com/dlink/service/impl/StudioServiceImpl.java
+14
-0
TaskServiceImpl.java
...src/main/java/com/dlink/service/impl/TaskServiceImpl.java
+14
-0
ExtractionUtils.java
.../apache/flink/table/types/extraction/ExtractionUtils.java
+947
-0
ExtractionUtils.java
.../apache/flink/table/types/extraction/ExtractionUtils.java
+947
-0
ExtractionUtils.java
.../apache/flink/table/types/extraction/ExtractionUtils.java
+985
-0
ExtractionUtils.java
.../apache/flink/table/types/extraction/ExtractionUtils.java
+986
-0
ClassEntity.java
dlink-common/src/main/java/com/dlink/pool/ClassEntity.java
+42
-0
ClassPool.java
dlink-common/src/main/java/com/dlink/pool/ClassPool.java
+61
-0
pom.xml
dlink-core/pom.xml
+5
-0
JobManager.java
dlink-core/src/main/java/com/dlink/job/JobManager.java
+23
-0
CustomStringJavaCompiler.java
...c/main/java/com/dlink/utils/CustomStringJavaCompiler.java
+166
-0
UDFUtil.java
dlink-core/src/main/java/com/dlink/utils/UDFUtil.java
+43
-0
index.less
...omponents/Studio/StudioRightTool/StudioUDFInfo/index.less
+9
-0
index.tsx
...components/Studio/StudioRightTool/StudioUDFInfo/index.tsx
+56
-0
index.tsx
dlink-web/src/components/Studio/StudioRightTool/index.tsx
+10
-1
Welcome.tsx
dlink-web/src/pages/Welcome.tsx
+3
-0
quickstart.md
docs/en-US/guide/quickstart.md
+1
-1
quickstart.md
docs/guide/quickstart.md
+1
-1
No files found.
README.md
View file @
2a2d4f6b
...
@@ -57,12 +57,12 @@ Dinky(原 Dlink):
...
@@ -57,12 +57,12 @@ Dinky(原 Dlink):
| | | 支持 yarn application 模式下 FlinkSQL 提交 | 0.4.0 |
| | | 支持 yarn application 模式下 FlinkSQL 提交 | 0.4.0 |
| | | 支持 kubernetes session 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 kubernetes session 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 kubernetes application 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 kubernetes application 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 UDF Java 方言Local模式在线编写、调试、动态加载 | 0.5.0 |
| | Flink 作业 | 支持 yarn application 模式下 Jar 提交 | 0.4.0 |
| | Flink 作业 | 支持 yarn application 模式下 Jar 提交 | 0.4.0 |
| | | 支持 k8s application 模式下 Jar 提交 | 0.5.0 |
| | | 支持 k8s application 模式下 Jar 提交 | 0.5.0 |
| | | 支持 作业 Cancel | 0.4.0 |
| | | 支持 作业 Cancel | 0.4.0 |
| | | 支持 作业 SavePoint 的 Cancel、Stop、Trigger | 0.4.0 |
| | | 支持 作业 SavePoint 的 Cancel、Stop、Trigger | 0.4.0 |
| | | 新增 作业自动从 SavePoint 恢复机制(包含最近、最早、指定一次) | 0.4.0 |
| | | 新增 作业自动从 SavePoint 恢复机制(包含最近、最早、指定一次) | 0.4.0 |
| | | 新增 UDF java方言代码的开发 | 0.5.0 |
| | Flink 集群 | 支持 查看已注册集群的作业列表与运维 | 0.4.0 |
| | Flink 集群 | 支持 查看已注册集群的作业列表与运维 | 0.4.0 |
| | | 新增 自动注册 Yarn 创建的集群 | 0.4.0 |
| | | 新增 自动注册 Yarn 创建的集群 | 0.4.0 |
| | SQL | 新增 外部数据源的 SQL 校验 | 0.5.0 |
| | SQL | 新增 外部数据源的 SQL 校验 | 0.5.0 |
...
...
dlink-admin/src/main/java/com/dlink/service/TaskService.java
View file @
2a2d4f6b
...
@@ -24,4 +24,6 @@ public interface TaskService extends ISuperService<Task> {
...
@@ -24,4 +24,6 @@ public interface TaskService extends ISuperService<Task> {
List
<
Task
>
listFlinkSQLEnv
();
List
<
Task
>
listFlinkSQLEnv
();
String
exportSql
(
Integer
id
);
String
exportSql
(
Integer
id
);
Task
getUDFByClassName
(
String
className
);
}
}
dlink-admin/src/main/java/com/dlink/service/impl/StudioServiceImpl.java
View file @
2a2d4f6b
...
@@ -27,6 +27,7 @@ import com.dlink.session.SessionConfig;
...
@@ -27,6 +27,7 @@ import com.dlink.session.SessionConfig;
import
com.dlink.session.SessionInfo
;
import
com.dlink.session.SessionInfo
;
import
com.dlink.session.SessionPool
;
import
com.dlink.session.SessionPool
;
import
com.dlink.utils.RunTimeUtil
;
import
com.dlink.utils.RunTimeUtil
;
import
com.dlink.utils.UDFUtil
;
import
com.fasterxml.jackson.core.JsonProcessingException
;
import
com.fasterxml.jackson.core.JsonProcessingException
;
import
com.fasterxml.jackson.databind.JsonNode
;
import
com.fasterxml.jackson.databind.JsonNode
;
import
com.fasterxml.jackson.databind.ObjectMapper
;
import
com.fasterxml.jackson.databind.ObjectMapper
;
...
@@ -89,6 +90,7 @@ public class StudioServiceImpl implements StudioService {
...
@@ -89,6 +90,7 @@ public class StudioServiceImpl implements StudioService {
if
(!
config
.
isUseSession
())
{
if
(!
config
.
isUseSession
())
{
config
.
setAddress
(
clusterService
.
buildEnvironmentAddress
(
config
.
isUseRemote
(),
studioExecuteDTO
.
getClusterId
()));
config
.
setAddress
(
clusterService
.
buildEnvironmentAddress
(
config
.
isUseRemote
(),
studioExecuteDTO
.
getClusterId
()));
}
}
initUDF
(
config
,
studioExecuteDTO
.
getStatement
());
JobManager
jobManager
=
JobManager
.
build
(
config
);
JobManager
jobManager
=
JobManager
.
build
(
config
);
JobResult
jobResult
=
jobManager
.
executeSql
(
studioExecuteDTO
.
getStatement
());
JobResult
jobResult
=
jobManager
.
executeSql
(
studioExecuteDTO
.
getStatement
());
RunTimeUtil
.
recovery
(
jobManager
);
RunTimeUtil
.
recovery
(
jobManager
);
...
@@ -152,6 +154,7 @@ public class StudioServiceImpl implements StudioService {
...
@@ -152,6 +154,7 @@ public class StudioServiceImpl implements StudioService {
if
(!
config
.
isUseSession
())
{
if
(!
config
.
isUseSession
())
{
config
.
setAddress
(
clusterService
.
buildEnvironmentAddress
(
config
.
isUseRemote
(),
studioExecuteDTO
.
getClusterId
()));
config
.
setAddress
(
clusterService
.
buildEnvironmentAddress
(
config
.
isUseRemote
(),
studioExecuteDTO
.
getClusterId
()));
}
}
initUDF
(
config
,
studioExecuteDTO
.
getStatement
());
JobManager
jobManager
=
JobManager
.
buildPlanMode
(
config
);
JobManager
jobManager
=
JobManager
.
buildPlanMode
(
config
);
return
jobManager
.
explainSql
(
studioExecuteDTO
.
getStatement
()).
getSqlExplainResults
();
return
jobManager
.
explainSql
(
studioExecuteDTO
.
getStatement
()).
getSqlExplainResults
();
}
}
...
@@ -317,4 +320,15 @@ public class StudioServiceImpl implements StudioService {
...
@@ -317,4 +320,15 @@ public class StudioServiceImpl implements StudioService {
}
}
return
false
;
return
false
;
}
}
private
void
initUDF
(
JobConfig
config
,
String
statement
){
if
(!
GatewayType
.
LOCAL
.
equalsValue
(
config
.
getType
())){
return
;
}
List
<
String
>
udfClassNameList
=
JobManager
.
getUDFClassName
(
statement
);
for
(
String
item
:
udfClassNameList
){
Task
task
=
taskService
.
getUDFByClassName
(
item
);
JobManager
.
initUDF
(
item
,
task
.
getStatement
());
}
}
}
}
dlink-admin/src/main/java/com/dlink/service/impl/TaskServiceImpl.java
View file @
2a2d4f6b
...
@@ -15,6 +15,7 @@ import com.dlink.job.JobResult;
...
@@ -15,6 +15,7 @@ import com.dlink.job.JobResult;
import
com.dlink.mapper.TaskMapper
;
import
com.dlink.mapper.TaskMapper
;
import
com.dlink.model.*
;
import
com.dlink.model.*
;
import
com.dlink.service.*
;
import
com.dlink.service.*
;
import
com.dlink.utils.CustomStringJavaCompiler
;
import
org.springframework.beans.factory.annotation.Autowired
;
import
org.springframework.beans.factory.annotation.Autowired
;
import
org.springframework.beans.factory.annotation.Value
;
import
org.springframework.beans.factory.annotation.Value
;
import
org.springframework.stereotype.Service
;
import
org.springframework.stereotype.Service
;
...
@@ -100,6 +101,11 @@ public class TaskServiceImpl extends SuperServiceImpl<TaskMapper, Task> implemen
...
@@ -100,6 +101,11 @@ public class TaskServiceImpl extends SuperServiceImpl<TaskMapper, Task> implemen
@Override
@Override
public
boolean
saveOrUpdateTask
(
Task
task
)
{
public
boolean
saveOrUpdateTask
(
Task
task
)
{
if
(
Asserts
.
isNotNullString
(
task
.
getDialect
())
&&
Dialect
.
JAVA
.
equalsVal
(
task
.
getDialect
())
&&
Asserts
.
isNotNullString
(
task
.
getStatement
())
){
CustomStringJavaCompiler
compiler
=
new
CustomStringJavaCompiler
(
task
.
getStatement
());
task
.
setSavePointPath
(
compiler
.
getFullClassName
());
}
if
(
task
.
getId
()
!=
null
)
{
if
(
task
.
getId
()
!=
null
)
{
this
.
updateById
(
task
);
this
.
updateById
(
task
);
if
(
task
.
getStatement
()
!=
null
)
{
if
(
task
.
getStatement
()
!=
null
)
{
...
@@ -151,6 +157,14 @@ public class TaskServiceImpl extends SuperServiceImpl<TaskMapper, Task> implemen
...
@@ -151,6 +157,14 @@ public class TaskServiceImpl extends SuperServiceImpl<TaskMapper, Task> implemen
}
}
}
}
@Override
public
Task
getUDFByClassName
(
String
className
)
{
Task
task
=
getOne
(
new
QueryWrapper
<
Task
>().
eq
(
"dialect"
,
"Java"
).
eq
(
"enabled"
,
1
).
eq
(
"save_point_path"
,
className
));
Assert
.
check
(
task
);
task
.
setStatement
(
statementService
.
getById
(
task
.
getId
()).
getStatement
());
return
task
;
}
private
JobConfig
buildJobConfig
(
Task
task
){
private
JobConfig
buildJobConfig
(
Task
task
){
boolean
isJarTask
=
isJarTask
(
task
);
boolean
isJarTask
=
isJarTask
(
task
);
if
(!
isJarTask
&&
Asserts
.
isNotNull
(
task
.
getEnvId
())){
if
(!
isJarTask
&&
Asserts
.
isNotNull
(
task
.
getEnvId
())){
...
...
dlink-client/dlink-client-1.11/src/main/java/org/apache/flink/table/types/extraction/ExtractionUtils.java
0 → 100644
View file @
2a2d4f6b
This diff is collapsed.
Click to expand it.
dlink-client/dlink-client-1.12/src/main/java/org/apache/flink/table/types/extraction/ExtractionUtils.java
0 → 100644
View file @
2a2d4f6b
This diff is collapsed.
Click to expand it.
dlink-client/dlink-client-1.13/src/main/java/org/apache/flink/table/types/extraction/ExtractionUtils.java
0 → 100644
View file @
2a2d4f6b
This diff is collapsed.
Click to expand it.
dlink-client/dlink-client-1.14/src/main/java/org/apache/flink/table/types/extraction/ExtractionUtils.java
0 → 100644
View file @
2a2d4f6b
This diff is collapsed.
Click to expand it.
dlink-common/src/main/java/com/dlink/pool/ClassEntity.java
0 → 100644
View file @
2a2d4f6b
package
com
.
dlink
.
pool
;
import
com.dlink.assertion.Asserts
;
import
lombok.Getter
;
import
lombok.Setter
;
/**
* ClassEntity
*
* @author wenmo
* @since 2022/1/12 23:52
*/
@Getter
@Setter
public
class
ClassEntity
{
private
String
name
;
private
String
code
;
private
byte
[]
classByte
;
public
ClassEntity
(
String
name
,
String
code
)
{
this
.
name
=
name
;
this
.
code
=
code
;
}
public
ClassEntity
(
String
name
,
String
code
,
byte
[]
classByte
)
{
this
.
name
=
name
;
this
.
code
=
code
;
this
.
classByte
=
classByte
;
}
public
static
ClassEntity
build
(
String
name
,
String
code
){
return
new
ClassEntity
(
name
,
code
);
}
public
boolean
equals
(
ClassEntity
entity
)
{
if
(
Asserts
.
isEquals
(
name
,
entity
.
getName
())
&&
Asserts
.
isEquals
(
code
,
entity
.
getCode
())){
return
true
;
}
else
{
return
false
;
}
}
}
dlink-common/src/main/java/com/dlink/pool/ClassPool.java
0 → 100644
View file @
2a2d4f6b
package
com
.
dlink
.
pool
;
import
java.util.List
;
import
java.util.Vector
;
/**
* ClassPool
*
* @author wenmo
* @since 2022/1/12 23:52
*/
public
class
ClassPool
{
private
static
volatile
List
<
ClassEntity
>
classList
=
new
Vector
<>();
public
static
boolean
exist
(
String
name
)
{
for
(
ClassEntity
executorEntity
:
classList
)
{
if
(
executorEntity
.
getName
().
equals
(
name
))
{
return
true
;
}
}
return
false
;
}
public
static
boolean
exist
(
ClassEntity
entity
)
{
for
(
ClassEntity
executorEntity
:
classList
)
{
if
(
executorEntity
.
equals
(
entity
))
{
return
true
;
}
}
return
false
;
}
public
static
Integer
push
(
ClassEntity
executorEntity
){
if
(
exist
(
executorEntity
.
getName
())){
remove
(
executorEntity
.
getName
());
}
classList
.
add
(
executorEntity
);
return
classList
.
size
();
}
public
static
Integer
remove
(
String
name
)
{
int
count
=
classList
.
size
();
for
(
int
i
=
0
;
i
<
classList
.
size
();
i
++)
{
if
(
name
.
equals
(
classList
.
get
(
i
).
getName
()))
{
classList
.
remove
(
i
);
break
;
}
}
return
count
-
classList
.
size
();
}
public
static
ClassEntity
get
(
String
name
)
{
for
(
ClassEntity
executorEntity
:
classList
)
{
if
(
executorEntity
.
getName
().
equals
(
name
))
{
return
executorEntity
;
}
}
return
null
;
}
}
dlink-core/pom.xml
View file @
2a2d4f6b
...
@@ -35,6 +35,11 @@
...
@@ -35,6 +35,11 @@
<groupId>
cn.hutool
</groupId>
<groupId>
cn.hutool
</groupId>
<artifactId>
hutool-all
</artifactId>
<artifactId>
hutool-all
</artifactId>
</dependency>
</dependency>
<dependency>
<groupId>
org.codehaus.groovy
</groupId>
<artifactId>
groovy
</artifactId>
<version>
3.0.9
</version>
</dependency>
<dependency>
<dependency>
<groupId>
junit
</groupId>
<groupId>
junit
</groupId>
<artifactId>
junit
</artifactId>
<artifactId>
junit
</artifactId>
...
...
dlink-core/src/main/java/com/dlink/job/JobManager.java
View file @
2a2d4f6b
...
@@ -19,6 +19,8 @@ import com.dlink.gateway.result.TestResult;
...
@@ -19,6 +19,8 @@ import com.dlink.gateway.result.TestResult;
import
com.dlink.interceptor.FlinkInterceptor
;
import
com.dlink.interceptor.FlinkInterceptor
;
import
com.dlink.model.SystemConfiguration
;
import
com.dlink.model.SystemConfiguration
;
import
com.dlink.parser.SqlType
;
import
com.dlink.parser.SqlType
;
import
com.dlink.pool.ClassEntity
;
import
com.dlink.pool.ClassPool
;
import
com.dlink.result.*
;
import
com.dlink.result.*
;
import
com.dlink.session.ExecutorEntity
;
import
com.dlink.session.ExecutorEntity
;
import
com.dlink.session.SessionConfig
;
import
com.dlink.session.SessionConfig
;
...
@@ -26,6 +28,7 @@ import com.dlink.session.SessionInfo;
...
@@ -26,6 +28,7 @@ import com.dlink.session.SessionInfo;
import
com.dlink.session.SessionPool
;
import
com.dlink.session.SessionPool
;
import
com.dlink.trans.Operations
;
import
com.dlink.trans.Operations
;
import
com.dlink.utils.SqlUtil
;
import
com.dlink.utils.SqlUtil
;
import
com.dlink.utils.UDFUtil
;
import
com.fasterxml.jackson.databind.node.ObjectNode
;
import
com.fasterxml.jackson.databind.node.ObjectNode
;
import
org.apache.flink.configuration.CoreOptions
;
import
org.apache.flink.configuration.CoreOptions
;
import
org.apache.flink.configuration.DeploymentOptions
;
import
org.apache.flink.configuration.DeploymentOptions
;
...
@@ -43,6 +46,8 @@ import java.util.ArrayList;
...
@@ -43,6 +46,8 @@ import java.util.ArrayList;
import
java.util.Collections
;
import
java.util.Collections
;
import
java.util.List
;
import
java.util.List
;
import
java.util.Map
;
import
java.util.Map
;
import
java.util.regex.Matcher
;
import
java.util.regex.Pattern
;
/**
/**
* JobManager
* JobManager
...
@@ -491,4 +496,22 @@ public class JobManager {
...
@@ -491,4 +496,22 @@ public class JobManager {
sb
.
append
(
statement
);
sb
.
append
(
statement
);
return
sb
.
toString
();
return
sb
.
toString
();
}
}
public
static
List
<
String
>
getUDFClassName
(
String
statement
){
Pattern
pattern
=
Pattern
.
compile
(
"function (.*?)'(.*?)'"
,
Pattern
.
CASE_INSENSITIVE
);
Matcher
matcher
=
pattern
.
matcher
(
statement
);
List
<
String
>
classNameList
=
new
ArrayList
<>();
while
(
matcher
.
find
())
{
classNameList
.
add
(
matcher
.
group
(
2
));
}
return
classNameList
;
}
public
static
void
initUDF
(
String
className
,
String
code
){
if
(
ClassPool
.
exist
(
ClassEntity
.
build
(
className
,
code
))){
UDFUtil
.
initClassLoader
(
className
);
}
else
{
UDFUtil
.
buildClass
(
code
);
}
}
}
}
dlink-core/src/main/java/com/dlink/utils/CustomStringJavaCompiler.java
0 → 100644
View file @
2a2d4f6b
package
com
.
dlink
.
utils
;
import
javax.tools.*
;
import
java.io.ByteArrayOutputStream
;
import
java.io.IOException
;
import
java.io.OutputStream
;
import
java.net.URI
;
import
java.util.Arrays
;
import
java.util.List
;
import
java.util.Map
;
import
java.util.concurrent.ConcurrentHashMap
;
import
java.util.regex.Matcher
;
import
java.util.regex.Pattern
;
/**
* CustomStringJavaCompiler
*
* @author wenmo
* @since 2021/12/28 22:46
*/
public
class
CustomStringJavaCompiler
{
//类全名
private
String
fullClassName
;
private
String
sourceCode
;
//存放编译之后的字节码(key:类全名,value:编译之后输出的字节码)
private
Map
<
String
,
ByteJavaFileObject
>
javaFileObjectMap
=
new
ConcurrentHashMap
<>();
//获取java的编译器
private
JavaCompiler
compiler
=
ToolProvider
.
getSystemJavaCompiler
();
//存放编译过程中输出的信息
private
DiagnosticCollector
<
JavaFileObject
>
diagnosticsCollector
=
new
DiagnosticCollector
<>();
//编译耗时(单位ms)
private
long
compilerTakeTime
;
public
String
getFullClassName
()
{
return
fullClassName
;
}
public
ByteJavaFileObject
getJavaFileObjectMap
(
String
name
)
{
return
javaFileObjectMap
.
get
(
name
);
}
public
CustomStringJavaCompiler
(
String
sourceCode
)
{
this
.
sourceCode
=
sourceCode
;
this
.
fullClassName
=
getFullClassName
(
sourceCode
);
}
/**
* 编译字符串源代码,编译失败在 diagnosticsCollector 中获取提示信息
*
* @return true:编译成功 false:编译失败
*/
public
boolean
compiler
()
{
long
startTime
=
System
.
currentTimeMillis
();
//标准的内容管理器,更换成自己的实现,覆盖部分方法
StandardJavaFileManager
standardFileManager
=
compiler
.
getStandardFileManager
(
diagnosticsCollector
,
null
,
null
);
JavaFileManager
javaFileManager
=
new
StringJavaFileManage
(
standardFileManager
);
//构造源代码对象
JavaFileObject
javaFileObject
=
new
StringJavaFileObject
(
fullClassName
,
sourceCode
);
//获取一个编译任务
JavaCompiler
.
CompilationTask
task
=
compiler
.
getTask
(
null
,
javaFileManager
,
diagnosticsCollector
,
null
,
null
,
Arrays
.
asList
(
javaFileObject
));
//设置编译耗时
compilerTakeTime
=
System
.
currentTimeMillis
()
-
startTime
;
return
task
.
call
();
}
/**
* @return 编译信息(错误 警告)
*/
public
String
getCompilerMessage
()
{
StringBuilder
sb
=
new
StringBuilder
();
List
<
Diagnostic
<?
extends
JavaFileObject
>>
diagnostics
=
diagnosticsCollector
.
getDiagnostics
();
for
(
Diagnostic
diagnostic
:
diagnostics
)
{
sb
.
append
(
diagnostic
.
toString
()).
append
(
"\r\n"
);
}
return
sb
.
toString
();
}
public
long
getCompilerTakeTime
()
{
return
compilerTakeTime
;
}
/**
* 获取类的全名称
*
* @param sourceCode 源码
* @return 类的全名称
*/
public
static
String
getFullClassName
(
String
sourceCode
)
{
String
className
=
""
;
Pattern
pattern
=
Pattern
.
compile
(
"package\\s+\\S+\\s*;"
);
Matcher
matcher
=
pattern
.
matcher
(
sourceCode
);
if
(
matcher
.
find
())
{
className
=
matcher
.
group
().
replaceFirst
(
"package"
,
""
).
replace
(
";"
,
""
).
trim
()
+
"."
;
}
pattern
=
Pattern
.
compile
(
"class\\s+(\\S+)\\s+"
);
matcher
=
pattern
.
matcher
(
sourceCode
);
if
(
matcher
.
find
())
{
className
+=
matcher
.
group
(
1
).
trim
();
}
return
className
;
}
/**
* 自定义一个字符串的源码对象
*/
private
class
StringJavaFileObject
extends
SimpleJavaFileObject
{
//等待编译的源码字段
private
String
contents
;
//java源代码 => StringJavaFileObject对象 的时候使用
public
StringJavaFileObject
(
String
className
,
String
contents
)
{
super
(
URI
.
create
(
"string:///"
+
className
.
replaceAll
(
"\\."
,
"/"
)
+
Kind
.
SOURCE
.
extension
),
Kind
.
SOURCE
);
this
.
contents
=
contents
;
}
//字符串源码会调用该方法
@Override
public
CharSequence
getCharContent
(
boolean
ignoreEncodingErrors
)
throws
IOException
{
return
contents
;
}
}
/**
* 自定义一个编译之后的字节码对象
*/
public
class
ByteJavaFileObject
extends
SimpleJavaFileObject
{
//存放编译后的字节码
private
ByteArrayOutputStream
outPutStream
;
public
ByteJavaFileObject
(
String
className
,
Kind
kind
)
{
super
(
URI
.
create
(
"string:///"
+
className
.
replaceAll
(
"\\."
,
"/"
)
+
Kind
.
SOURCE
.
extension
),
kind
);
}
//StringJavaFileManage 编译之后的字节码输出会调用该方法(把字节码输出到outputStream)
@Override
public
OutputStream
openOutputStream
()
{
outPutStream
=
new
ByteArrayOutputStream
();
return
outPutStream
;
}
//在类加载器加载的时候需要用到
public
byte
[]
getCompiledBytes
()
{
return
outPutStream
.
toByteArray
();
}
}
/**
* 自定义一个JavaFileManage来控制编译之后字节码的输出位置
*/
private
class
StringJavaFileManage
extends
ForwardingJavaFileManager
{
StringJavaFileManage
(
JavaFileManager
fileManager
)
{
super
(
fileManager
);
}
//获取输出的文件对象,它表示给定位置处指定类型的指定类。
@Override
public
JavaFileObject
getJavaFileForOutput
(
Location
location
,
String
className
,
JavaFileObject
.
Kind
kind
,
FileObject
sibling
)
throws
IOException
{
ByteJavaFileObject
javaFileObject
=
new
ByteJavaFileObject
(
className
,
kind
);
javaFileObjectMap
.
put
(
className
,
javaFileObject
);
return
javaFileObject
;
}
}
}
dlink-core/src/main/java/com/dlink/utils/UDFUtil.java
0 → 100644
View file @
2a2d4f6b
package
com
.
dlink
.
utils
;
import
com.dlink.pool.ClassEntity
;
import
com.dlink.pool.ClassPool
;
import
groovy.lang.GroovyClassLoader
;
import
org.codehaus.groovy.control.CompilerConfiguration
;
/**
* UDFUtil
*
* @author wenmo
* @since 2021/12/27 23:25
*/
public
class
UDFUtil
{
public
static
void
buildClass
(
String
code
){
CustomStringJavaCompiler
compiler
=
new
CustomStringJavaCompiler
(
code
);
boolean
res
=
compiler
.
compiler
();
if
(
res
)
{
String
className
=
compiler
.
getFullClassName
();
byte
[]
compiledBytes
=
compiler
.
getJavaFileObjectMap
(
className
).
getCompiledBytes
();
ClassPool
.
push
(
new
ClassEntity
(
className
,
code
,
compiledBytes
));
System
.
out
.
println
(
"编译成功"
);
System
.
out
.
println
(
"compilerTakeTime:"
+
compiler
.
getCompilerTakeTime
());
initClassLoader
(
className
);
}
else
{
System
.
out
.
println
(
"编译失败"
);
System
.
out
.
println
(
compiler
.
getCompilerMessage
());
}
}
public
static
void
initClassLoader
(
String
name
){
ClassEntity
classEntity
=
ClassPool
.
get
(
name
);
ClassLoader
contextClassLoader
=
Thread
.
currentThread
().
getContextClassLoader
();
CompilerConfiguration
config
=
new
CompilerConfiguration
();
config
.
setSourceEncoding
(
"UTF-8"
);
GroovyClassLoader
groovyClassLoader
=
new
GroovyClassLoader
(
contextClassLoader
,
config
);
groovyClassLoader
.
setShouldRecompile
(
true
);
groovyClassLoader
.
defineClass
(
classEntity
.
getName
(),
classEntity
.
getClassByte
());
Thread
.
currentThread
().
setContextClassLoader
(
groovyClassLoader
);
// Class<?> clazz = groovyClassLoader.parseClass(codeSource,"com.dlink.ud.udf.SubstringFunction");
}
}
dlink-web/src/components/Studio/StudioRightTool/StudioUDFInfo/index.less
0 → 100644
View file @
2a2d4f6b
@import '~antd/es/style/themes/default.less';
.form_setting{
padding-left: 10px;
}
.form_item{
margin-bottom: 5px;
}
dlink-web/src/components/Studio/StudioRightTool/StudioUDFInfo/index.tsx
0 → 100644
View file @
2a2d4f6b
import
{
connect
}
from
"umi"
;
import
{
StateType
}
from
"@/pages/FlinkSqlStudio/model"
;
import
{
Form
,
Switch
,
Row
,
Col
,
Tooltip
,
Button
,
Input
}
from
"antd"
;
import
{
InfoCircleOutlined
,
MinusSquareOutlined
}
from
"@ant-design/icons"
;
import
styles
from
"./index.less"
;
import
{
useEffect
}
from
"react"
;
import
{
JarStateType
}
from
"@/pages/Jar/model"
;
import
{
Scrollbars
}
from
"react-custom-scrollbars"
;
const
StudioUDFInfo
=
(
props
:
any
)
=>
{
const
{
current
,
form
,
toolHeight
}
=
props
;
useEffect
(()
=>
{
form
.
setFieldsValue
(
current
.
task
);
},
[
current
.
task
]);
return
(
<>
<
Row
>
<
Col
span=
{
24
}
>
<
div
style=
{
{
float
:
"right"
}
}
>
<
Tooltip
title=
"最小化"
>
<
Button
type=
"text"
icon=
{
<
MinusSquareOutlined
/>
}
/>
</
Tooltip
>
</
div
>
</
Col
>
</
Row
>
<
Scrollbars
style=
{
{
height
:
(
toolHeight
-
32
)}
}
>
<
Form
form=
{
form
}
layout=
"vertical"
className=
{
styles
.
form_setting
}
>
<
Row
>
<
Col
span=
{
24
}
>
<
Form
.
Item
label=
"类名"
className=
{
styles
.
form_item
}
name=
"savePointPath"
>
<
Input
readOnly=
{
true
}
placeholder=
"自动识别"
/>
</
Form
.
Item
>
</
Col
>
</
Row
>
</
Form
>
</
Scrollbars
>
</>
);
};
export
default
connect
(({
Studio
,
Jar
}:
{
Studio
:
StateType
,
Jar
:
JarStateType
})
=>
({
current
:
Studio
.
current
,
toolHeight
:
Studio
.
toolHeight
,
}))(
StudioUDFInfo
);
dlink-web/src/components/Studio/StudioRightTool/index.tsx
View file @
2a2d4f6b
...
@@ -8,6 +8,7 @@ import StudioSetting from "./StudioSetting";
...
@@ -8,6 +8,7 @@ import StudioSetting from "./StudioSetting";
import
StudioSavePoint
from
"./StudioSavePoint"
;
import
StudioSavePoint
from
"./StudioSavePoint"
;
import
StudioEnvSetting
from
"./StudioEnvSetting"
;
import
StudioEnvSetting
from
"./StudioEnvSetting"
;
import
StudioSqlConfig
from
"./StudioSqlConfig"
;
import
StudioSqlConfig
from
"./StudioSqlConfig"
;
import
StudioUDFInfo
from
"./StudioUDFInfo"
;
import
{
DIALECT
,
isSql
}
from
"@/components/Studio/conf"
;
import
{
DIALECT
,
isSql
}
from
"@/components/Studio/conf"
;
const
{
TabPane
}
=
Tabs
;
const
{
TabPane
}
=
Tabs
;
...
@@ -26,7 +27,7 @@ const StudioRightTool = (props:any) => {
...
@@ -26,7 +27,7 @@ const StudioRightTool = (props:any) => {
return
renderEnvContent
();
return
renderEnvContent
();
}
}
if
(
DIALECT
.
JAVA
===
current
.
task
.
dialect
){
if
(
DIALECT
.
JAVA
===
current
.
task
.
dialect
){
return
undefined
;
return
renderUDFContent
()
;
}
}
return
renderFlinkSqlContent
();
return
renderFlinkSqlContent
();
};
};
...
@@ -47,6 +48,14 @@ const StudioRightTool = (props:any) => {
...
@@ -47,6 +48,14 @@ const StudioRightTool = (props:any) => {
</>)
</>)
};
};
const
renderUDFContent
=
()
=>
{
return
(<>
<
TabPane
tab=
{
<
span
><
SettingOutlined
/>
UDF信息
</
span
>
}
key=
"StudioUDFInfo"
>
<
StudioUDFInfo
form=
{
form
}
/>
</
TabPane
>
</>)
};
const
renderFlinkSqlContent
=
()
=>
{
const
renderFlinkSqlContent
=
()
=>
{
return
(<><
TabPane
tab=
{
<
span
><
SettingOutlined
/>
作业配置
</
span
>
}
key=
"StudioSetting"
>
return
(<><
TabPane
tab=
{
<
span
><
SettingOutlined
/>
作业配置
</
span
>
}
key=
"StudioSetting"
>
<
StudioSetting
form=
{
form
}
/>
<
StudioSetting
form=
{
form
}
/>
...
...
dlink-web/src/pages/Welcome.tsx
View file @
2a2d4f6b
...
@@ -547,6 +547,9 @@ export default (): React.ReactNode => {
...
@@ -547,6 +547,9 @@ export default (): React.ReactNode => {
<
li
>
<
li
>
<
Link
>
新增 快捷键保存、校验、美化
</
Link
>
<
Link
>
新增 快捷键保存、校验、美化
</
Link
>
</
li
>
</
li
>
<
li
>
<
Link
>
新增 UDF Java方言的Local模式的在线编写、调试、动态加载
</
Link
>
</
li
>
</
ul
>
</
ul
>
</
Paragraph
>
</
Paragraph
>
</
Timeline
.
Item
>
</
Timeline
.
Item
>
...
...
docs/en-US/guide/quickstart.md
View file @
2a2d4f6b
...
@@ -95,12 +95,12 @@ Dinky 通过已注册的集群配置来获取对应的 YarnClient 实例。对
...
@@ -95,12 +95,12 @@ Dinky 通过已注册的集群配置来获取对应的 YarnClient 实例。对
| | | 支持 yarn application 模式下 FlinkSQL 提交 | 0.4.0 |
| | | 支持 yarn application 模式下 FlinkSQL 提交 | 0.4.0 |
| | | 支持 kubernetes session 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 kubernetes session 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 kubernetes application 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 kubernetes application 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 UDF Java 方言Local模式在线编写、调试、动态加载 | 0.5.0 |
| | Flink 作业 | 支持 yarn application 模式下 Jar 提交 | 0.4.0 |
| | Flink 作业 | 支持 yarn application 模式下 Jar 提交 | 0.4.0 |
| | | 支持 k8s application 模式下 Jar 提交 | 0.5.0 |
| | | 支持 k8s application 模式下 Jar 提交 | 0.5.0 |
| | | 支持 作业 Cancel | 0.4.0 |
| | | 支持 作业 Cancel | 0.4.0 |
| | | 支持 作业 SavePoint 的 Cancel、Stop、Trigger | 0.4.0 |
| | | 支持 作业 SavePoint 的 Cancel、Stop、Trigger | 0.4.0 |
| | | 新增 作业自动从 SavePoint 恢复机制(包含最近、最早、指定一次) | 0.4.0 |
| | | 新增 作业自动从 SavePoint 恢复机制(包含最近、最早、指定一次) | 0.4.0 |
| | | 新增 UDF java方言代码的开发 | 0.5.0 |
| | Flink 集群 | 支持 查看已注册集群的作业列表与运维 | 0.4.0 |
| | Flink 集群 | 支持 查看已注册集群的作业列表与运维 | 0.4.0 |
| | | 新增 自动注册 Yarn 创建的集群 | 0.4.0 |
| | | 新增 自动注册 Yarn 创建的集群 | 0.4.0 |
| | SQL | 新增 外部数据源的 SQL 校验 | 0.5.0 |
| | SQL | 新增 外部数据源的 SQL 校验 | 0.5.0 |
...
...
docs/guide/quickstart.md
View file @
2a2d4f6b
...
@@ -95,12 +95,12 @@ Dinky 通过已注册的集群配置来获取对应的 YarnClient 实例。对
...
@@ -95,12 +95,12 @@ Dinky 通过已注册的集群配置来获取对应的 YarnClient 实例。对
| | | 支持 yarn application 模式下 FlinkSQL 提交 | 0.4.0 |
| | | 支持 yarn application 模式下 FlinkSQL 提交 | 0.4.0 |
| | | 支持 kubernetes session 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 kubernetes session 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 kubernetes application 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 kubernetes application 模式下 FlinkSQL 提交 | 0.5.0 |
| | | 支持 UDF Java 方言Local模式在线编写、调试、动态加载 | 0.5.0 |
| | Flink 作业 | 支持 yarn application 模式下 Jar 提交 | 0.4.0 |
| | Flink 作业 | 支持 yarn application 模式下 Jar 提交 | 0.4.0 |
| | | 支持 k8s application 模式下 Jar 提交 | 0.5.0 |
| | | 支持 k8s application 模式下 Jar 提交 | 0.5.0 |
| | | 支持 作业 Cancel | 0.4.0 |
| | | 支持 作业 Cancel | 0.4.0 |
| | | 支持 作业 SavePoint 的 Cancel、Stop、Trigger | 0.4.0 |
| | | 支持 作业 SavePoint 的 Cancel、Stop、Trigger | 0.4.0 |
| | | 新增 作业自动从 SavePoint 恢复机制(包含最近、最早、指定一次) | 0.4.0 |
| | | 新增 作业自动从 SavePoint 恢复机制(包含最近、最早、指定一次) | 0.4.0 |
| | | 新增 UDF java方言代码的开发 | 0.5.0 |
| | Flink 集群 | 支持 查看已注册集群的作业列表与运维 | 0.4.0 |
| | Flink 集群 | 支持 查看已注册集群的作业列表与运维 | 0.4.0 |
| | | 新增 自动注册 Yarn 创建的集群 | 0.4.0 |
| | | 新增 自动注册 Yarn 创建的集群 | 0.4.0 |
| | SQL | 新增 外部数据源的 SQL 校验 | 0.5.0 |
| | SQL | 新增 外部数据源的 SQL 校验 | 0.5.0 |
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment