Skip to content

Commit 664b9d0

Browse files
ChenYi015AranshnpgretzCian911jacobsalway
authored
Release v2.1.0 (#2354)
* Allow setting automountServiceAccountToken (#2298) * Allow setting automountServiceAccountToken on workloads and serviceAccounts Signed-off-by: Aran Shavit <[email protected]> * update helm docs Signed-off-by: Aran Shavit <[email protected]> --------- Signed-off-by: Aran Shavit <[email protected]> (cherry picked from commit 515d805) * Fix: executor container security context does not work (#2306) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 171e429) * Fix: should not add emptyDir sizeLimit conf if it is nil (#2305) Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 763682d) * Allow the Controller and Webhook Containers to run with the securityContext: readOnlyRootfilesystem: true (#2282) * create a tmp dir for the controller to write Spark artifacts to and set the controller to readOnlyRootFilesystem Signed-off-by: Nick Gretzon <[email protected]> * mount a dir for the webhook container to generate its certificates in and set readOnlyRootFilesystem: true for the webhook pod Signed-off-by: Nick Gretzon <[email protected]> * update the securityContext in the controller deployment test Signed-off-by: Nick Gretzon <[email protected]> * update securityContext of the webhook container in the deployment_test Signed-off-by: Nick Gretzon <[email protected]> * update README Signed-off-by: Nick Gretzon <[email protected]> * remove -- so comments are not rendered in the README.md Signed-off-by: Nick Gretzon <[email protected]> * recreate README.md after removal of comments for volumes and volumeMounts Signed-off-by: Nick Gretzon <[email protected]> * make indentation for volumes and volumeMounts consistent with rest of values.yaml Signed-off-by: Nick Gretzon <[email protected]> * Revert "make indentation for volumes and volumeMounts consistent with rest of values.yaml" This reverts commit dba97fc. Signed-off-by: Nick Gretzon <[email protected]> * fix indentation in webhook and controller deployment templates for volumes and volumeMounts Signed-off-by: Nick Gretzon <[email protected]> * Update charts/spark-operator-chart/values.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/values.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/values.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/values.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/templates/controller/deployment.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/templates/controller/deployment.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/templates/webhook/deployment.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * Update charts/spark-operator-chart/templates/webhook/deployment.yaml Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> * add additional securityContext to the controller deployment_test.yaml Signed-off-by: Nick Gretzon <[email protected]> --------- Signed-off-by: Nick Gretzon <[email protected]> Signed-off-by: Nicholas Gretzon <[email protected]> Co-authored-by: Yi Chen <[email protected]> (cherry picked from commit 72107fd) * Fix: should not add emptyDir sizeLimit conf on executor pods if it is nil (#2316) Signed-off-by: Cian Gallagher <[email protected]> (cherry picked from commit 2999546) * Bump `volcano.sh/apis` to 1.10.0 (#2320) Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 22e4fb8) * Truncate UI service name if over 63 characters (#2311) * Truncate UI service name if over 63 characters Signed-off-by: Jacob Salway <[email protected]> * Also truncate ingress name Signed-off-by: Jacob Salway <[email protected]> --------- Signed-off-by: Jacob Salway <[email protected]> (cherry picked from commit 43c1888) * Bump aquasecurity/trivy-action from 0.28.0 to 0.29.0 (#2332) Bumps [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action) from 0.28.0 to 0.29.0. - [Release notes](https://github.com/aquasecurity/trivy-action/releases) - [Commits](aquasecurity/trivy-action@0.28.0...0.29.0) --- updated-dependencies: - dependency-name: aquasecurity/trivy-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 270b09e) * Bump github.com/onsi/ginkgo/v2 from 2.20.2 to 2.22.0 (#2335) Bumps [github.com/onsi/ginkgo/v2](https://github.com/onsi/ginkgo) from 2.20.2 to 2.22.0. - [Release notes](https://github.com/onsi/ginkgo/releases) - [Changelog](https://github.com/onsi/ginkgo/blob/master/CHANGELOG.md) - [Commits](onsi/ginkgo@v2.20.2...v2.22.0) --- updated-dependencies: - dependency-name: github.com/onsi/ginkgo/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 40423d5) * The webhook-key-name command-line param isn't taking effect (#2344) Signed-off-by: C. H. Afzal <[email protected]> (cherry picked from commit a261523) * Robustness to driver pod taking time to create (#2315) * Retry after driver pod now found if recent submission Signed-off-by: Thomas Newton <[email protected]> * Add a test Signed-off-by: Thomas Newton <[email protected]> * Make grace period configurable Signed-off-by: Thomas Newton <[email protected]> * Update test Signed-off-by: Thomas Newton <[email protected]> * Add an extra test with the driver pod Signed-off-by: Thomas Newton <[email protected]> * Separate context to create and delete the driver pod Signed-off-by: Thomas Newton <[email protected]> * Tidy Signed-off-by: Thomas Newton <[email protected]> * Autoformat Signed-off-by: Thomas Newton <[email protected]> * Update error message Signed-off-by: Thomas Newton <[email protected]> * Add helm paramater Signed-off-by: Thomas Newton <[email protected]> * Update internal/controller/sparkapplication/controller.go Co-authored-by: Yi Chen <[email protected]> Signed-off-by: Thomas Newton <[email protected]> * Newlines between helm tests Signed-off-by: Thomas Newton <[email protected]> --------- Signed-off-by: Thomas Newton <[email protected]> Co-authored-by: Yi Chen <[email protected]> (cherry picked from commit d815e78) * Use NSS_WRAPPER_PASSWD instead of /etc/passwd as in spark-operator image entrypoint.sh (#2312) Signed-off-by: Aakcht <[email protected]> (cherry picked from commit 5dd91c4) * Move sparkctl to cmd directory (#2347) * Move spark-operator Signed-off-by: Yi Chen <[email protected]> * Move sparkctl to cmd directory Signed-off-by: Yi Chen <[email protected]> * Remove unnecessary app package/directory Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> (cherry picked from commit 2375a30) * Spark Operator Official Release v2.1.0 Signed-off-by: Yi Chen <[email protected]> --------- Signed-off-by: Yi Chen <[email protected]> Co-authored-by: Aran Shavit <[email protected]> Co-authored-by: Nicholas Gretzon <[email protected]> Co-authored-by: Cian (Keen) Gallagher <[email protected]> Co-authored-by: Jacob Salway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: C. H. Afzal <[email protected]> Co-authored-by: Thomas Newton <[email protected]> Co-authored-by: Aakcht <[email protected]>
1 parent 1ff61a4 commit 664b9d0

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+387
-137
lines changed

‎.github/workflows/trivy-image-scanning.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ jobs:
1515
run: make print-IMAGE >> $GITHUB_ENV
1616

1717
- name: trivy scan for github security tab
18-
uses: aquasecurity/trivy-action@0.28.0
18+
uses: aquasecurity/trivy-action@0.29.0
1919
with:
2020
image-ref: '${{ env.IMAGE }}'
2121
format: 'sarif'

‎Makefile

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -173,12 +173,12 @@ override LDFLAGS += \
173173
.PHONY: build-operator
174174
build-operator: ## Build Spark operator.
175175
echo "Building spark-operator binary..."
176-
go build -o $(SPARK_OPERATOR) -ldflags '${LDFLAGS}' cmd/main.go
176+
go build -o $(SPARK_OPERATOR) -ldflags '${LDFLAGS}' cmd/operator/main.go
177177

178178
.PHONY: build-sparkctl
179179
build-sparkctl: ## Build sparkctl binary.
180180
echo "Building sparkctl binary..."
181-
CGO_ENABLED=0 go build -o $(SPARKCTL) -buildvcs=false sparkctl/main.go
181+
CGO_ENABLED=0 go build -o $(SPARKCTL) -buildvcs=false cmd/sparkctl/main.go
182182

183183
.PHONY: install-sparkctl
184184
install-sparkctl: build-sparkctl ## Install sparkctl binary.
@@ -191,7 +191,7 @@ clean: ## Clean spark-operator and sparktcl binaries.
191191
rm -f $(SPARKCTL)
192192

193193
.PHONY: build-api-docs
194-
build-api-docs: gen-crd-api-reference-docs ## Build api documentaion.
194+
build-api-docs: gen-crd-api-reference-docs ## Build api documentation.
195195
$(GEN_CRD_API_REFERENCE_DOCS) \
196196
-config hack/api-docs/config.json \
197197
-api-dir github.com/kubeflow/spark-operator/api/v1beta2 \

‎VERSION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
v2.1.0-rc.0
1+
v2.1.0

‎charts/spark-operator-chart/Chart.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,9 @@ name: spark-operator
2020

2121
description: A Helm chart for Spark on Kubernetes operator.
2222

23-
version: 2.1.0-rc.0
23+
version: 2.1.0
2424

25-
appVersion: 2.1.0-rc.0
25+
appVersion: 2.1.0
2626

2727
keywords:
2828
- apache spark

‎charts/spark-operator-chart/README.md

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# spark-operator
22

3-
![Version: 2.1.0-rc.0](https://img.shields.io/badge/Version-2.1.0--rc.0-informational?style=flat-square) ![AppVersion: 2.1.0-rc.0](https://img.shields.io/badge/AppVersion-2.1.0--rc.0-informational?style=flat-square)
3+
![Version: 2.1.0](https://img.shields.io/badge/Version-2.1.0-informational?style=flat-square) ![AppVersion: 2.1.0](https://img.shields.io/badge/AppVersion-2.1.0-informational?style=flat-square)
44

55
A Helm chart for Spark on Kubernetes operator.
66

@@ -86,6 +86,7 @@ See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall) for command docum
8686
| controller.replicas | int | `1` | Number of replicas of controller. |
8787
| controller.workers | int | `10` | Reconcile concurrency, higher values might increase memory usage. |
8888
| controller.logLevel | string | `"info"` | Configure the verbosity of logging, can be one of `debug`, `info`, `error`. |
89+
| controller.driverPodCreationGracePeriod | string | `"10s"` | Grace period after a successful spark-submit when driver pod not found errors will be retried. Useful if the driver pod can take some time to be created. |
8990
| controller.maxTrackedExecutorPerApp | int | `1000` | Specifies the maximum number of Executor pods that can be tracked by the controller per SparkApplication. |
9091
| controller.uiService.enable | bool | `true` | Specifies whether to create service for Spark web UI. |
9192
| controller.uiIngress.enable | bool | `false` | Specifies whether to create ingress for Spark web UI. `controller.uiService.enable` must be `true` to enable ingress. |
@@ -97,11 +98,12 @@ See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall) for command docum
9798
| controller.serviceAccount.create | bool | `true` | Specifies whether to create a service account for the controller. |
9899
| controller.serviceAccount.name | string | `""` | Optional name for the controller service account. |
99100
| controller.serviceAccount.annotations | object | `{}` | Extra annotations for the controller service account. |
101+
| controller.serviceAccount.automountServiceAccountToken | bool | `true` | Auto-mount service account token to the controller pods. |
100102
| controller.rbac.create | bool | `true` | Specifies whether to create RBAC resources for the controller. |
101103
| controller.rbac.annotations | object | `{}` | Extra annotations for the controller RBAC resources. |
102104
| controller.labels | object | `{}` | Extra labels for controller pods. |
103105
| controller.annotations | object | `{}` | Extra annotations for controller pods. |
104-
| controller.volumes | list | `[]` | Volumes for controller pods. |
106+
| controller.volumes | list | `[{"emptyDir":{"sizeLimit":"1Gi"},"name":"tmp"}]` | Volumes for controller pods. |
105107
| controller.nodeSelector | object | `{}` | Node selector for controller pods. |
106108
| controller.affinity | object | `{}` | Affinity for controller pods. |
107109
| controller.tolerations | list | `[]` | List of node taints to tolerate for controller pods. |
@@ -110,9 +112,9 @@ See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall) for command docum
110112
| controller.topologySpreadConstraints | list | `[]` | Topology spread constraints rely on node labels to identify the topology domain(s) that each Node is in. Ref: [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/). The labelSelector field in topology spread constraint will be set to the selector labels for controller pods if not specified. |
111113
| controller.env | list | `[]` | Environment variables for controller containers. |
112114
| controller.envFrom | list | `[]` | Environment variable sources for controller containers. |
113-
| controller.volumeMounts | list | `[]` | Volume mounts for controller containers. |
115+
| controller.volumeMounts | list | `[{"mountPath":"/tmp","name":"tmp","readOnly":false}]` | Volume mounts for controller containers. |
114116
| controller.resources | object | `{}` | Pod resource requests and limits for controller containers. Note, that each job submission will spawn a JVM within the controller pods using "/usr/local/openjdk-11/bin/java -Xmx128m". Kubernetes may kill these Java processes at will to enforce resource limits. When that happens, you will see the following error: 'failed to run spark-submit for SparkApplication [...]: signal: killed' - when this happens, you may want to increase memory limits. |
115-
| controller.securityContext | object | `{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true}` | Security context for controller containers. |
117+
| controller.securityContext | object | `{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"privileged":false,"readOnlyRootFilesystem":true,"runAsNonRoot":true}` | Security context for controller containers. |
116118
| controller.sidecars | list | `[]` | Sidecar containers for controller pods. |
117119
| controller.podDisruptionBudget.enable | bool | `false` | Specifies whether to create pod disruption budget for controller. Ref: [Specifying a Disruption Budget for your Application](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) |
118120
| controller.podDisruptionBudget.minAvailable | int | `1` | The number of pods that must be available. Require `controller.replicas` to be greater than 1 |
@@ -134,12 +136,13 @@ See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall) for command docum
134136
| webhook.serviceAccount.create | bool | `true` | Specifies whether to create a service account for the webhook. |
135137
| webhook.serviceAccount.name | string | `""` | Optional name for the webhook service account. |
136138
| webhook.serviceAccount.annotations | object | `{}` | Extra annotations for the webhook service account. |
139+
| webhook.serviceAccount.automountServiceAccountToken | bool | `true` | Auto-mount service account token to the webhook pods. |
137140
| webhook.rbac.create | bool | `true` | Specifies whether to create RBAC resources for the webhook. |
138141
| webhook.rbac.annotations | object | `{}` | Extra annotations for the webhook RBAC resources. |
139142
| webhook.labels | object | `{}` | Extra labels for webhook pods. |
140143
| webhook.annotations | object | `{}` | Extra annotations for webhook pods. |
141144
| webhook.sidecars | list | `[]` | Sidecar containers for webhook pods. |
142-
| webhook.volumes | list | `[]` | Volumes for webhook pods. |
145+
| webhook.volumes | list | `[{"emptyDir":{"sizeLimit":"500Mi"},"name":"serving-certs"}]` | Volumes for webhook pods. |
143146
| webhook.nodeSelector | object | `{}` | Node selector for webhook pods. |
144147
| webhook.affinity | object | `{}` | Affinity for webhook pods. |
145148
| webhook.tolerations | list | `[]` | List of node taints to tolerate for webhook pods. |
@@ -148,15 +151,16 @@ See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall) for command docum
148151
| webhook.topologySpreadConstraints | list | `[]` | Topology spread constraints rely on node labels to identify the topology domain(s) that each Node is in. Ref: [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/). The labelSelector field in topology spread constraint will be set to the selector labels for webhook pods if not specified. |
149152
| webhook.env | list | `[]` | Environment variables for webhook containers. |
150153
| webhook.envFrom | list | `[]` | Environment variable sources for webhook containers. |
151-
| webhook.volumeMounts | list | `[]` | Volume mounts for webhook containers. |
154+
| webhook.volumeMounts | list | `[{"mountPath":"/etc/k8s-webhook-server/serving-certs","name":"serving-certs","readOnly":false,"subPath":"serving-certs"}]` | Volume mounts for webhook containers. |
152155
| webhook.resources | object | `{}` | Pod resource requests and limits for webhook pods. |
153-
| webhook.securityContext | object | `{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"privileged":false,"runAsNonRoot":true}` | Security context for webhook containers. |
156+
| webhook.securityContext | object | `{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"privileged":false,"readOnlyRootFilesystem":true,"runAsNonRoot":true}` | Security context for webhook containers. |
154157
| webhook.podDisruptionBudget.enable | bool | `false` | Specifies whether to create pod disruption budget for webhook. Ref: [Specifying a Disruption Budget for your Application](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) |
155158
| webhook.podDisruptionBudget.minAvailable | int | `1` | The number of pods that must be available. Require `webhook.replicas` to be greater than 1 |
156159
| spark.jobNamespaces | list | `["default"]` | List of namespaces where to run spark jobs. If empty string is included, all namespaces will be allowed. Make sure the namespaces have already existed. |
157160
| spark.serviceAccount.create | bool | `true` | Specifies whether to create a service account for spark applications. |
158161
| spark.serviceAccount.name | string | `""` | Optional name for the spark service account. |
159162
| spark.serviceAccount.annotations | object | `{}` | Optional annotations for the spark service account. |
163+
| spark.serviceAccount.automountServiceAccountToken | bool | `true` | Auto-mount service account token to the spark applications pods. |
160164
| spark.rbac.create | bool | `true` | Specifies whether to create RBAC resources for spark applications. |
161165
| spark.rbac.annotations | object | `{}` | Optional annotations for the spark application RBAC resources. |
162166
| prometheus.metrics.enable | bool | `true` | Specifies whether to enable prometheus metrics scraping. |

‎charts/spark-operator-chart/templates/controller/deployment.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,9 @@ spec:
100100
{{- if .Values.controller.workqueueRateLimiter.maxDelay.enable }}
101101
- --workqueue-ratelimiter-max-delay={{ .Values.controller.workqueueRateLimiter.maxDelay.duration }}
102102
{{- end }}
103+
{{- if .Values.controller.driverPodCreationGracePeriod }}
104+
- --driver-pod-creation-grace-period={{ .Values.controller.driverPodCreationGracePeriod }}
105+
{{- end }}
103106
{{- if .Values.controller.maxTrackedExecutorPerApp }}
104107
- --max-tracked-executor-per-app={{ .Values.controller.maxTrackedExecutorPerApp }}
105108
{{- end }}
@@ -171,6 +174,7 @@ spec:
171174
priorityClassName: {{ . }}
172175
{{- end }}
173176
serviceAccountName: {{ include "spark-operator.controller.serviceAccountName" . }}
177+
automountServiceAccountToken: {{ .Values.controller.serviceAccount.automountServiceAccountToken }}
174178
{{- with .Values.controller.podSecurityContext }}
175179
securityContext:
176180
{{- toYaml . | nindent 8 }}

‎charts/spark-operator-chart/templates/controller/serviceaccount.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ limitations under the License.
1717
{{- if .Values.controller.serviceAccount.create }}
1818
apiVersion: v1
1919
kind: ServiceAccount
20+
automountServiceAccountToken: {{ .Values.controller.serviceAccount.automountServiceAccountToken }}
2021
metadata:
2122
name: {{ include "spark-operator.controller.serviceAccountName" . }}
2223
namespace: {{ .Release.Namespace }}

‎charts/spark-operator-chart/templates/spark/serviceaccount.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ limitations under the License.
2121
---
2222
apiVersion: v1
2323
kind: ServiceAccount
24+
automountServiceAccountToken: {{ $.Values.spark.serviceAccount.automountServiceAccountToken }}
2425
metadata:
2526
name: {{ include "spark-operator.spark.serviceAccountName" $ }}
2627
namespace: {{ $jobNamespace }}

‎charts/spark-operator-chart/templates/webhook/deployment.yaml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ spec:
9494
{{- end }}
9595
{{- with .Values.webhook.volumeMounts }}
9696
volumeMounts:
97-
{{- toYaml . | nindent 10 }}
97+
{{- toYaml . | nindent 8 }}
9898
{{- end }}
9999
{{- with .Values.webhook.resources }}
100100
resources:
@@ -123,7 +123,7 @@ spec:
123123
{{- end }}
124124
{{- with .Values.webhook.volumes }}
125125
volumes:
126-
{{- toYaml . | nindent 8 }}
126+
{{- toYaml . | nindent 6 }}
127127
{{- end }}
128128
{{- with .Values.webhook.nodeSelector }}
129129
nodeSelector:
@@ -141,6 +141,7 @@ spec:
141141
priorityClassName: {{ . }}
142142
{{- end }}
143143
serviceAccountName: {{ include "spark-operator.webhook.serviceAccountName" . }}
144+
automountServiceAccountToken: {{ .Values.webhook.serviceAccount.automountServiceAccountToken }}
144145
{{- with .Values.webhook.podSecurityContext }}
145146
securityContext:
146147
{{- toYaml . | nindent 8 }}

‎charts/spark-operator-chart/templates/webhook/serviceaccount.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ limitations under the License.
1818
{{- if .Values.webhook.serviceAccount.create -}}
1919
apiVersion: v1
2020
kind: ServiceAccount
21+
automountServiceAccountToken: {{ .Values.webhook.serviceAccount.automountServiceAccountToken }}
2122
metadata:
2223
name: {{ include "spark-operator.webhook.serviceAccountName" . }}
2324
namespace: {{ .Release.Namespace }}

‎charts/spark-operator-chart/tests/controller/deployment_test.yaml

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -355,16 +355,30 @@ tests:
355355
set:
356356
controller:
357357
securityContext:
358+
readOnlyRootFilesystem: true
358359
runAsUser: 1000
359360
runAsGroup: 2000
360361
fsGroup: 3000
362+
allowPrivilegeEscalation: false
363+
capabilities:
364+
drop:
365+
- ALL
366+
runAsNonRoot: true
367+
privileged: false
361368
asserts:
362369
- equal:
363370
path: spec.template.spec.containers[0].securityContext
364371
value:
372+
readOnlyRootFilesystem: true
365373
runAsUser: 1000
366374
runAsGroup: 2000
367375
fsGroup: 3000
376+
allowPrivilegeEscalation: false
377+
capabilities:
378+
drop:
379+
- ALL
380+
runAsNonRoot: true
381+
privileged: false
368382

369383
- it: Should add sidecars if `controller.sidecars` is set
370384
set:
@@ -637,6 +651,16 @@ tests:
637651
- notContains:
638652
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
639653
content: --workqueue-ratelimiter-max-delay=1h
654+
655+
- it: Should contain `driver-pod-creation-grace-period` arg if `controller.driverPodCreationGracePeriod` is set
656+
set:
657+
controller:
658+
driverPodCreationGracePeriod: 30s
659+
asserts:
660+
- contains:
661+
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
662+
content: --driver-pod-creation-grace-period=30s
663+
640664
- it: Should contain `--max-tracked-executor-per-app` arg if `controller.maxTrackedExecutorPerApp` is set
641665
set:
642666
controller:

‎charts/spark-operator-chart/tests/webhook/deployment_test.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -299,10 +299,14 @@ tests:
299299
set:
300300
webhook:
301301
securityContext:
302+
readOnlyRootFilesystem: true
302303
runAsUser: 1000
303304
runAsGroup: 2000
304305
fsGroup: 3000
305306
asserts:
307+
- equal:
308+
path: spec.template.spec.containers[0].securityContext.readOnlyRootFilesystem
309+
value: true
306310
- equal:
307311
path: spec.template.spec.containers[0].securityContext.runAsUser
308312
value: 1000

‎charts/spark-operator-chart/values.yaml

Lines changed: 33 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,9 @@ controller:
5151
# -- Configure the verbosity of logging, can be one of `debug`, `info`, `error`.
5252
logLevel: info
5353

54+
# -- Grace period after a successful spark-submit when driver pod not found errors will be retried. Useful if the driver pod can take some time to be created.
55+
driverPodCreationGracePeriod: 10s
56+
5457
# -- Specifies the maximum number of Executor pods that can be tracked by the controller per SparkApplication.
5558
maxTrackedExecutorPerApp: 1000
5659

@@ -87,6 +90,8 @@ controller:
8790
name: ""
8891
# -- Extra annotations for the controller service account.
8992
annotations: {}
93+
# -- Auto-mount service account token to the controller pods.
94+
automountServiceAccountToken: true
9095

9196
rbac:
9297
# -- Specifies whether to create RBAC resources for the controller.
@@ -105,7 +110,11 @@ controller:
105110
# key2: value2
106111

107112
# -- Volumes for controller pods.
108-
volumes: []
113+
volumes:
114+
# Create a tmp directory to write Spark artifacts to for deployed Spark apps.
115+
- name: tmp
116+
emptyDir:
117+
sizeLimit: 1Gi
109118

110119
# -- Node selector for controller pods.
111120
nodeSelector: {}
@@ -141,7 +150,11 @@ controller:
141150
envFrom: []
142151

143152
# -- Volume mounts for controller containers.
144-
volumeMounts: []
153+
volumeMounts:
154+
# Mount a tmp directory to write Spark artifacts to for deployed Spark apps.
155+
- name: tmp
156+
mountPath: "/tmp"
157+
readOnly: false
145158

146159
# -- Pod resource requests and limits for controller containers.
147160
# Note, that each job submission will spawn a JVM within the controller pods using "/usr/local/openjdk-11/bin/java -Xmx128m".
@@ -157,6 +170,7 @@ controller:
157170

158171
# -- Security context for controller containers.
159172
securityContext:
173+
readOnlyRootFilesystem: true
160174
privileged: false
161175
allowPrivilegeEscalation: false
162176
runAsNonRoot: true
@@ -231,6 +245,8 @@ webhook:
231245
name: ""
232246
# -- Extra annotations for the webhook service account.
233247
annotations: {}
248+
# -- Auto-mount service account token to the webhook pods.
249+
automountServiceAccountToken: true
234250

235251
rbac:
236252
# -- Specifies whether to create RBAC resources for the webhook.
@@ -252,7 +268,11 @@ webhook:
252268
sidecars: []
253269

254270
# -- Volumes for webhook pods.
255-
volumes: []
271+
volumes:
272+
# Create a dir for the webhook to generate its certificates in.
273+
- name: serving-certs
274+
emptyDir:
275+
sizeLimit: 500Mi
256276

257277
# -- Node selector for webhook pods.
258278
nodeSelector: {}
@@ -288,7 +308,13 @@ webhook:
288308
envFrom: []
289309

290310
# -- Volume mounts for webhook containers.
291-
volumeMounts: []
311+
volumeMounts:
312+
# Mount a dir for the webhook to generate its certificates in.
313+
- name: serving-certs
314+
mountPath: /etc/k8s-webhook-server/serving-certs
315+
subPath: serving-certs
316+
readOnly: false
317+
292318

293319
# -- Pod resource requests and limits for webhook pods.
294320
resources: {}
@@ -301,6 +327,7 @@ webhook:
301327

302328
# -- Security context for webhook containers.
303329
securityContext:
330+
readOnlyRootFilesystem: true
304331
privileged: false
305332
allowPrivilegeEscalation: false
306333
runAsNonRoot: true
@@ -331,6 +358,8 @@ spark:
331358
name: ""
332359
# -- Optional annotations for the spark service account.
333360
annotations: {}
361+
# -- Auto-mount service account token to the spark applications pods.
362+
automountServiceAccountToken: true
334363

335364
rbac:
336365
# -- Specifies whether to create RBAC resources for spark applications.

‎cmd/main.go

Lines changed: 0 additions & 31 deletions
This file was deleted.

0 commit comments

Comments
 (0)