feat: remove shared of openmldb dependencies for openmldb-batchjob (#3849)

* Remove shared of openmldb dependencies for openmldb-batchjob

* Add shade for protobuf in openmldb-batchjob

* Change log to logger in taskmanager

* Update deploy doc for spark distribution

* fix minor typos

---------

Co-authored-by: Siqi Wang <[email protected]>

Author: tobegit3hub

docs/en/deploy/install_deploy.md

@@ -24,7 +24,7 @@ If you need to deploy ZooKeeper and TaskManager, you need a Java runtime environ
 
 Servers needs Java 1.8 or above.
 
-Zookeeper Client 3.4.14 requires `Java 1.7` - `Java 13`. Java SDK depends on it, so it should use the same Java version, don't run in higher version. If you wish to use zkCli, please use `Java 1.8` or `Java 11`.
+Zookeeper Client 3.4.14 requires `Java 1.7` - `Java 13`. Java SDK depends on the same client, so it should use the same Java version, not a higher version. If you wish to use zkCli, please use `Java 1.8` or `Java 11`.
 
 ### Hardware
 
@@ -195,7 +195,7 @@ The environment variables are defined in `conf/openmldb-env.sh`, as shown in the
 | OPENMLDB_VERSION                  | 0.8.5                                                   | OpenMLDB version                                             |
 | OPENMLDB_MODE                     | standalone                                              | standalone or cluster                                        |
 | OPENMLDB_HOME                     | root directory of the release folder                    | openmldb root directory                                      |
-| SPARK_HOME                        | $OPENMLDB_HOME/spark                                    | openmldb spark root directory，If the directory does not exist,  it will be downloaded automatically.|
+| SPARK_HOME                        | $OPENMLDB_HOME/spark                                    | Spark root directory, if the directory does not exist,  it will be downloaded automatically.|
 | OPENMLDB_TABLET_PORT              | 10921                                                   | TabletServer default port                                    |
 | OPENMLDB_NAMESERVER_PORT          | 7527                                                    | NameServer default port                                      |
 | OPENMLDB_TASKMANAGER_PORT         | 9902                                                    | taskmanager default port                                     |
@@ -205,7 +205,7 @@ The environment variables are defined in `conf/openmldb-env.sh`, as shown in the
 | OPENMLDB_ZK_CLUSTER               | auto derived from `[zookeeper]` section in `conf/hosts` | ZooKeeper cluster address                                    |
 | OPENMLDB_ZK_ROOT_PATH             | /openmldb                                               | OpenMLDB root directory in ZooKeeper                         |
 | OPENMLDB_ZK_CLUSTER_CLIENT_PORT   | 2181                                                    | ZooKeeper client port, the client port in zoo.cfg            |
-| OPENMLDB_ZK_CLUSTER_PEER_PORT     | 2888                                                    | ZooKeeper peer port，the first port in settings like "server.1=zoo1:2888:3888" in zoo.cfg |
+| OPENMLDB_ZK_CLUSTER_PEER_PORT     | 2888                                                    | ZooKeeper peer port, the first port in settings like "server.1=zoo1:2888:3888" in zoo.cfg |
 | OPENMLDB_ZK_CLUSTER_ELECTION_PORT | 3888                                                    | ZooKeeper election port, the second port in settings like "server.1=zoo1:2888:3888" in zoo.cfg |
 
 ### Node Configuration
@@ -252,7 +252,7 @@ If multiple TaskManager instances are deployed on distinct machines, the configu
 bash sbin/init_env.sh
 ```
 Note:
-- This script requires root execution. Other scripts does not require root privileges.
+- This script requires root execution. Other scripts do not require root privileges.
 - The script only modifies limit configurations, disabling swap and THP.
 
 ### Deployment
@@ -360,7 +360,7 @@ Enter `quit` or `Ctrl+C` to exit the zk client.
 
 ### Deploy TabletServer
 
-Note that at least two TabletServer need to be deployed, otherwise errors may occur.
+Note that at least two TabletServers need to be deployed, otherwise errors may occur.
 
 **1. Download the OpenMLDB deployment package**
 
@@ -439,7 +439,7 @@ mv openmldb-0.8.5-linux openmldb-tablet-0.8.5-2
 cd openmldb-tablet-0.8.5-2
 ```
 
-Modify the configuration again and start the TabletServer. Note that if all TabletServers are on the same machine, use different port numbers to avoid "Fail to listen" error in the log (`logs/tablet.WARNING`).
+Modify the configuration again and start the TabletServer. Note that if all TabletServers are on the same machine, use different port numbers to avoid the "Fail to listen" error in the log (`logs/tablet.WARNING`).
 
 **Note:**
 
@@ -608,9 +608,11 @@ The results should include information about all TabletServer and NameServer tha
 
 You can have only one TaskManager, but if you require high availability, you can deploy multiple TaskManagers, taking care to avoid IP and port conflicts. If the TaskManager master node experiences a failure, a slave node will automatically recover and replace the master node. Clients can continue accessing the TaskManager service without any modifications.
 
-**1. Download the OpenMLDB deployment package and Spark distribution for feature engineering optimization**
+**1. Download the OpenMLDB deployment package and Spark distribution **
 
-Spark distribution：
+Download the Spark distribution from the [Spark official website](https://spark.apache.org/downloads.html)。 Then unzip it and set the `SPARK_HOME` environment variable.
+
+Alternatively, use the OpenMLDB Spark distribution.
 
 ```shell
 wget https://github.com/4paradigm/spark/releases/download/v3.2.1-openmldb0.8.5/spark-3.2.1-bin-openmldbspark.tgz

docs/zh/deploy/install_deploy.md

@@ -192,10 +192,10 @@ cd openmldb-0.8.5-linux
 
 | 环境变量                         | 默认值                   | 定义                                                                                                                                          |
 | -------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
-| OPENMLDB_VERSION                 | 0.8.5                    | OpenMLDB版本，主要用于spark下载，一般不改动。                                                                                                 |
+| OPENMLDB_VERSION                 | 0.8.5                    | OpenMLDB版本，主要用于Spark下载，一般不改动。                                                                                                 |
 | OPENMLDB_MODE                    | cluster                  | standalone或者cluster                                                                                                                         |
 | OPENMLDB_HOME                    | 当前发行版的根目录       | openmldb发行版根目录，不则使用当前根目录，也就是openmldb-0.8.5-linux所在目录。                                                                |
-| SPARK_HOME                       | $OPENMLDB_HOME/spark     | openmldb spark发行版根目录，如果该目录不存在，自动从网上下载。**此路径也将成为TaskManager运行机器上的Spark安装目录。**                        |
+| SPARK_HOME                       | $OPENMLDB_HOME/spark     | Spark发行版根目录，如果该目录不存在，自动从网上下载。**此路径也将成为TaskManager运行机器上的Spark安装目录。**                        |
 | RUNNER_EXISTING_SPARK_HOME       |                          | 配置此项，运行TaskManager的机器将使用该Spark环境，将不下载、部署OpenMLDB Spark发行版。                                                        |
 | OPENMLDB_USE_EXISTING_ZK_CLUSTER | false                    | 是否使用已经运行的ZooKeeper集群。如果是`true`，将跳过ZooKeeper集群的部署与管理。                                                              |
 | OPENMLDB_ZK_HOME                 | $OPENMLDB_HOME/zookeeper | ZooKeeper发行版根目录，如果该目录不存在，自动从网上下载。                                                                                     |
@@ -208,7 +208,7 @@ cd openmldb-0.8.5-linux
 通常来讲，需要确认以下几点：
 - ZooKeeper集群地址，如果使用已有ZooKeeper集群，需要配置`OPENMLDB_USE_EXISTING_ZK_CLUSTER=true`，并配置`OPENMLDB_ZK_CLUSTER`。（如果在`conf/hosts`中配置外部ZK集群，请注释标注其不受sbin部署影响，避免混乱。）
 - 需要此工具部署ZooKeeper集群时，在`conf/hosts`中配置`[zookeeper]`。填写多个ZooKeeper节点，即部署ZooKeeper集群，无需额外配置。
-- Spark环境，如果需要使用运行机器上已有的Spark环境，需要配置`RUNNER_EXISTING_SPARK_HOME`（地址为TaskManager运行机器上的路径）。如果部署机器存在Spark环境，并想要在TaskManager机器上使用此套环境，可配置`SPARK_HOME`（部署到TaskManager机器同名路径上）。`SPARK_HOME`不进行配置时，将自动下载、使用OpenMLDB Spark发行版。
+- Spark环境，如果需要使用运行机器上已有的Spark环境，需要配置`RUNNER_EXISTING_SPARK_HOME`（地址为TaskManager运行机器上的路径）。如果部署机器存在Spark环境，并想要在TaskManager机器上使用此套环境，可配置`SPARK_HOME`（部署到TaskManager机器同名路径上）。`SPARK_HOME`不进行配置时，将自动下载、使用特定Spark发行版。
 
 #### 默认端口
 | 环境变量                  | 默认值 | 定义                 |
@@ -628,9 +628,14 @@ curl http://<apiserver_ip>:<port>/dbs/foo -X POST -d'{"mode":"online","sql":"sho
 
 TaskManager 可以只存在一台，如果你需要高可用性，可以部署多 TaskManager ，需要注意避免IP端口冲突。如果 TaskManager 主节点出现故障，从节点将自动恢复故障取代主节点，客户端无需任何修改可继续访问 TaskManager 服务。
 
-**1. 下载 OpenMLDB 部署包和面向特征工程优化的 Spark 发行版**
+**1. 下载 OpenMLDB 部署包和 Spark 发行版**
 
 Spark发行版：
+
+从 Spark 官网下载[Spark 发行版](https://spark.apache.org/downloads.html)，解压后配置`SPARK_HOME`环境变量。
+  
+或者使用 OpenMLDB Spark 发行版。
+
 ```shell
 wget https://github.com/4paradigm/spark/releases/download/v3.2.1-openmldb0.8.5/spark-3.2.1-bin-openmldbspark.tgz
 # 中国镜像地址：https://www.openmldb.com/download/v0.8.5/spark-3.2.1-bin-openmldbspark.tgz

java/openmldb-batchjob/pom.xml

@@ -23,6 +23,26 @@
   </properties>
 
   <dependencies>
+
+    <!-- OpenMLDB -->
+    <dependency>
+      <groupId>com.4paradigm.openmldb</groupId>
+      <artifactId>openmldb-native</artifactId>
+      <version>${variant.native.version}</version>
+    </dependency>
+
+    <dependency>
+      <groupId>com.4paradigm.openmldb</groupId>
+      <artifactId>openmldb-batch</artifactId>
+      <version>${project.parent.version}</version>
+      <exclusions>
+        <exclusion>
+          <groupId>org.apache.spark</groupId>
+          <artifactId>*</artifactId>
+        </exclusion>
+      </exclusions>
+    </dependency>
+
     <dependency>
       <groupId>junit</groupId>
       <artifactId>junit</artifactId>
@@ -64,27 +84,6 @@
       <scope>${spark.scope}</scope>
     </dependency>
 
-    <!-- OpenMLDB -->
-    <dependency>
-      <groupId>com.4paradigm.openmldb</groupId>
-      <artifactId>openmldb-native</artifactId>
-      <version>${variant.native.version}</version>
-      <scope>provided</scope>
-    </dependency>
-
-    <dependency>
-      <groupId>com.4paradigm.openmldb</groupId>
-      <artifactId>openmldb-batch</artifactId>
-      <version>${project.parent.version}</version>
-      <exclusions>
-        <exclusion>
-          <groupId>org.apache.spark</groupId>
-          <artifactId>*</artifactId>
-        </exclusion>
-      </exclusions>
-      <scope>provided</scope>
-    </dependency>
-
   </dependencies>
 
   <build>
@@ -129,10 +128,12 @@
               <goal>shade</goal>
             </goals>
             <configuration>
-              <artifactSet>
-                <excludes>
-                </excludes>
-              </artifactSet>
+              <relocations>
+                <relocation>
+                  <pattern>com.google.protobuf</pattern>
+                  <shadedPattern>shade.protobuf</shadedPattern>
+                </relocation>
+              </relocations>
             </configuration>
           </execution>
         </executions>

java/openmldb-taskmanager/src/main/java/com/_4paradigm/openmldb/taskmanager/server/impl/TaskManagerImpl.java

@@ -481,7 +481,7 @@ public TaskManager.SaveJobResultResponse SaveJobResult(TaskManager.SaveJobResult
         }
         // log if save failed
         if (!jobResultSaver.saveFile(request.getResultId(), request.getJsonData())) {
-            log.error("save job result failed(write to local file) for resultId: {}", request.getResultId());
+            logger.error("save job result failed(write to local file) for resultId: " + request.getResultId());
             return TaskManager.SaveJobResultResponse.newBuilder().setCode(StatusCode.FAILED)
                     .setMsg("save job result failed(write to local file)").build();
         }