0x00 生态版本

flink dinky hdfs
1.17.1 0.7.4 3.1.1

0x01 报错信息

安装Dinky0.7.4的时候,报Caused by: java.lang.NoClassDefFoundError: org/apache/flink/table/planner/plan/optimize/program/FlinkChainedProgram的错误,具体错误细节参考下面日志详情。

我是2023-10-16安装的dinky0.7.4的版本,由于安装过程报错日志没有保留下来,因此使用了该文中的日志:安装使用dinky0.7不成功,报错如下。说是缺少flink依赖包。请问缺少什么flink依赖包?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
[dlink] 2022-12-08 10:45:08 CST ERROR com.dlink.exception.WebExceptionHandler 97 unknownException - ERROR: org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: org/apache/flink/table/planner/plan/optimize/program/FlinkChainedProgram
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1086) ~[spring-webmvc-5.3.23.jar:5.3.23]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:964) ~[spring-webmvc-5.3.23.jar:5.3.23]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.23.jar:5.3.23]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.23.jar:5.3.23]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:696) ~[tomcat-embed-core-9.0.68.jar:4.0.FR]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.23.jar:5.3.23]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:779) ~[tomcat-embed-core-9.0.68.jar:4.0.FR]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.68.jar:9.0.68]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:124) ~[druid-1.2.8.jar:1.2.8]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.23.jar:5.3.23]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.23.jar:5.3.23]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.23.jar:5.3.23]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.23.jar:5.3.23]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.23.jar:5.3.23]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.23.jar:5.3.23]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.68.jar:9.0.68]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/table/planner/plan/optimize/program/FlinkChainedProgram
at com.dlink.utils.FlinkStreamProgramWithoutPhysical.buildProgram(FlinkStreamProgramWithoutPhysical.java:54) ~[dlink-client-1.16-0.7.0.jar:?]
at com.dlink.executor.CustomTableEnvironmentImpl.<init>(CustomTableEnvironmentImpl.java:112) ~[dlink-client-1.16-0.7.0.jar:?]
at com.dlink.executor.CustomTableEnvironmentImpl.create(CustomTableEnvironmentImpl.java:176) ~[dlink-client-1.16-0.7.0.jar:?]
at com.dlink.executor.CustomTableEnvironmentImpl.create(CustomTableEnvironmentImpl.java:116) ~[dlink-client-1.16-0.7.0.jar:?]
at com.dlink.executor.RemoteStreamExecutor.createCustomTableEnvironment(RemoteStreamExecutor.java:49) ~[dlink-executor-0.7.0.jar:?]
at com.dlink.executor.Executor.updateStreamExecutionEnvironment(Executor.java:201) ~[dlink-executor-0.7.0.jar:?]
at com.dlink.executor.Executor.initStreamExecutionEnvironment(Executor.java:195) ~[dlink-executor-0.7.0.jar:?]
at com.dlink.executor.Executor.init(Executor.java:169) ~[dlink-executor-0.7.0.jar:?]
at com.dlink.executor.RemoteStreamExecutor.<init>(RemoteStreamExecutor.java:44) ~[dlink-executor-0.7.0.jar:?]
at com.dlink.executor.Executor.buildRemoteExecutor(Executor.java:119) ~[dlink-executor-0.7.0.jar:?]
at com.dlink.job.JobManager.createExecutor(JobManager.java:202) ~[dlink-core-0.7.0.jar:?]
at com.dlink.job.JobManager.createExecutorWithSession(JobManager.java:223) ~[dlink-core-0.7.0.jar:?]
at com.dlink.job.JobManager.initUDF(JobManager.java:261) ~[dlink-core-0.7.0.jar:?]
at com.dlink.job.JobManager.init(JobManager.java:250) ~[dlink-core-0.7.0.jar:?]
at com.dlink.job.JobManager.build(JobManager.java:171) ~[dlink-core-0.7.0.jar:?]
at com.dlink.service.impl.StudioServiceImpl.executeFlinkSql(StudioServiceImpl.java:186) ~[dlink-admin-0.7.0.jar:?]
at com.dlink.service.impl.StudioServiceImpl.executeSql(StudioServiceImpl.java:173) ~[dlink-admin-0.7.0.jar:?]
at com.dlink.controller.StudioController.executeSql(StudioController.java:78) ~[dlink-admin-0.7.0.jar:?]
at com.dlink.controller.StudioController$$FastClassBySpringCGLIB$$e6483d87.invoke(<generated>) ~[dlink-admin-0.7.0.jar:?]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.23.jar:5.3.23]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.23.jar:5.3.23]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.23.jar:5.3.23]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar:5.3.23]
at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) ~[spring-aop-5.3.23.jar:5.3.23]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar:5.3.23]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar:5.3.23]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) ~[spring-aop-5.3.23.jar:5.3.23]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.23.jar:5.3.23]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.23.jar:5.3.23]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.23.jar:5.3.23]
at com.dlink.controller.StudioController$$EnhancerBySpringCGLIB$$84a184c1.executeSql(<generated>) ~[dlink-admin-0.7.0.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.23.jar:5.3.23]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.23.jar:5.3.23]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.23.jar:5.3.23]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.23.jar:5.3.23]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.23.jar:5.3.23]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.23.jar:5.3.23]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1071) ~[spring-webmvc-5.3.23.jar:5.3.23]
... 42 more

0x02 解决方法

$FLINK_HOME/libflink-table-planner-loader-1.17.1.jar移动到$FLINK_HOME/opt,并将$FLINK_HOME/opt中的flink-table-planner_2.12-1.17.1.jar移动到$FLINK_HOME/lib

特别注意:在flink/lib中所作的操作,必须在dinky/plugins/fink${FLINK_VERSION}/中做相同的操作,即保证两个目录中的jar包是一致的(除了部分特殊的jar)。

特别注意:在flink/lib中所作的操作,必须在dinky/plugins/fink${FLINK_VERSION}/中做相同的操作,即保证两个目录中的jar包是一致的(除了部分特殊的jar)。

特别注意:在flink/lib中所作的操作,必须在dinky/plugins/fink${FLINK_VERSION}/中做相同的操作,即保证两个目录中的jar包是一致的(除了部分特殊的jar)。

0x03 扩展:table Planner 和 Table Planner 加载器

从 Flink 1.15 开始,发行版包含两个 planner:

flink-table-planner_2.12-1.17.1.jar, 位于 /opt 目录, 包含查询计划器;
flink-table-planner-loader-1.17.1.jar, 位于 /lib 目录默认被加载, 包含隐藏在单独的 classpath 里的查询计划器 (您无法直接使用 io.apache.flink.table.planner 包)。
这两个 planner JAR 文件的代码功能相同,但打包方式不同。若使用第一个文件,您必须使用与其相同版本的 Scala;若使用第二个,由于 Scala 已经被打包进该文件里,您不需要考虑 Scala 版本问题。

默认情况下,发行版使用 flink-table-planner-loader。如果想使用内部查询计划器,您可以换掉 JAR 包(拷贝 flink-table-planner_2.12.jar 并复制到发行版的 /lib 目录)。请注意,此时会被限制用于 Flink 发行版的 Scala 版本。

这两个 planner 无法同时存在于 classpath,如果您在 /lib 目录同时加载他们,Table 任务将会失败。
在即将发布的 Flink 版本中,我们将停止在 Flink 发行版中发布 flink-table-planner_2.12 组件。我们强烈建议迁移您的作业/自定义连接器/格式以使用前述 API 模块,而不依赖此内部 planner。如果您需要 planner 中尚未被 API 模块暴露的一些功能,请与社区讨论。