Skip to content

Commit 9af4740

Browse files
kkulczakKacper Kulczak
authored and
Kacper Kulczak
committed
Update execute-kfp-pipelines-locally.md by adding example for interacting with Google Cloud services
Signed-off-by: Kacper Kulczak <[email protected]>
1 parent d116f2d commit 9af4740

File tree

1 file changed

+38
-0
lines changed

1 file changed

+38
-0
lines changed

content/en/docs/components/pipelines/user-guides/core-functions/execute-kfp-pipelines-locally.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ Local execution comes with several limitations:
2323
- Local execution does not feature optimizations and additional features such as caching, retry, etc. While these feature are important for production pipelines, they are less critical for a local testing environment. You will find that task methods like `.set_retry`, `.set_caching_options`, etc. have no effect locally.
2424
- Local execution makes simple assumptions about the resources available on your machine. Local execution does not support specifying resource requests/limits/affinities related to memory, cores, accelerators, etc. You will find that task methods like `.set_memory_limit`, `.set_memory_request`, `.set_accelerator_type` etc. have no effect locally.
2525
- Local execution doesn't support authentication mechanisms. If your component interacts with cloud resources or requires other privileged actions, you must test your pipeline in the cloud.
26+
- Exception: Note that the `SubprocessRunner` component does support Google Cloud authentication.
2627
- While local pipeline execution has full support for sequential and nested pipelines, it does not yet support `dsl.Condition`, `dsl.ParallelFor`, or `dsl.ExitHandler`.
2728

2829
## Basic Example
@@ -93,6 +94,43 @@ local.init(runner=...,
9394
pipeline_root='~/my/component/outputs')
9495
```
9596

97+
## Google Cloud Example
98+
99+
In the following example, we use `SubprocessRunner` to interact with Google Cloud services from the local machine.
100+
Before executing this script, you must authenticate with [Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc). You can use the following command: `gcloud auth application-default login`
101+
102+
```python
103+
# Before running this script setup Application Default Credentials
104+
# https://cloud.google.com/docs/authentication/provide-credentials-adc
105+
# with command `gcloud auth application-default login`
106+
# for authorization into Google Cloud
107+
from kfp import dsl, local
108+
109+
PROJECT_ID = "YOUR-PROJECT-ID"
110+
local.init(runner=local.SubprocessRunner())
111+
112+
@dsl.component(
113+
packages_to_install=["google-cloud-storage"]
114+
)
115+
def list_buckets(project_id: str) -> str:
116+
import json
117+
from google.cloud import storage
118+
storage_client = storage.Client(project=project_id)
119+
buckets = storage_client.list_buckets()
120+
return json.dumps([bucket.name for bucket in buckets])
121+
122+
@dsl.pipeline
123+
def gcs_list_pipeline(project_id: str) -> str:
124+
list_task = list_buckets(
125+
project_id=project_id,
126+
)
127+
return list_task.output
128+
129+
130+
if __name__ == "__main__":
131+
gcs_list_pipeline(project_id=PROJECT_ID)
132+
```
133+
96134
## Runner Types
97135

98136
Kubeflow pipelines has two local runners that you can use to execute your components and pipelines locally: `DockerRunner` and `SubprocessRunner`.

0 commit comments

Comments
 (0)