Skip to content

Commit 69bf69b

Browse files
smackeseyalangenfeld
authored andcommitted
[dg] Update python environment config (#29290)
## Summary & Motivation This PR makes several changes to `dg`'s venv management: - Change the configuration schema for setting a project python environment Previous: ```[tool.dg.project] python_environment = "active" OR python_environment = "persistent_uv" ``` Now: ```[tool.dg.project.python_environment] active = true OR uv_managed = true ``` - Change the default python environment for scaffolded projects from `uv_managed` to `active`. This means that scaffolding a new project no longer creates a venv/uv.lock by default. - Remove the `--skip-venv` option to project scaffolding commands. This no longer makes sense. We never create a venv when using `active`, and we always create a venv when using `uv_managed`. A consequence of the switch from scaffolding uv_managed to active python environment is that I needed to update all the guide snapshot tests to use `--python-environment uv_managed`. In a followup I will switch back the ones that should be using "active" mode, but adjusting the tutorials to use "active" should go in its own PR. Note a third `tool.dg.project.python_environment` option, `path`, will be added in a followup: ``` [tool.dg.project.python_environment] path = "/path/to/venv" ``` ## How I Tested These Changes Updated existing tests. ## Changelog - Changed configuration of project Python environments. The `tool.dg.project.python_environment` previously accepted a string, `"active"` or `"persistent_uv"`. Now it accepts a table with one of three keys: - `{active = true}`: equivalent of previous `"active"` - `{uv_managed = true}`: equivalent of previous `"persistent_uv"` - Changed the default python environment for newly scaffolded projects to `tool.dg.project.python_environment` to `{active = true}`. This means by default, no virtual environment or `uv.lock` will be created when scaffolding a new project (via `dg init` or `dg scaffold project`). You can pass `--python-environment uv_managed` for the old behavior. - Removed the `--skip-venv` flag on `dg scaffold project` and `dg init`.
1 parent ca061de commit 69bf69b

File tree

23 files changed

+245
-105
lines changed

23 files changed

+245
-105
lines changed

examples/docs_snippets/docs_snippets/guides/components/index/2-scaffold.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
dg scaffold project jaffle-platform
1+
dg scaffold project jaffle-platform --python-environment uv_managed
22

33
Creating a Dagster project at /.../jaffle-platform.
44
Scaffolded files for Dagster project at /.../jaffle-platform.

examples/docs_snippets/docs_snippets/guides/components/index/7-dg-list-plugins.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
dg list plugins
22

3+
Using /.../jaffle-platform/.venv/bin/dagster-components
34
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
45
┃ Plugin ┃ Objects ┃
56
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
dg scaffold component-type ShellCommand
22

3+
Using /.../my-component-library/.venv/bin/dagster-components
34
Creating a Dagster component type at /.../my-component-library/src/my_component_library/lib/shell_command.py.
45
Scaffolded files for Dagster component type at /.../my-component-library/src/my_component_library/lib/shell_command.py.

examples/docs_snippets/docs_snippets/guides/dg/configuring-dg/project-config.toml

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -30,17 +30,20 @@ code_location_name = "my-project"
3030
# (string, optional) Specifies the python environment to use when spawning subprocesses
3131
# for this project. Subprocesses are spawned whenever `dg` needs to list
3232
# available component types, check definitions, scaffold new components, etc.
33-
# The value must be one of:
34-
#
35-
# - "active" (default): use the currently active Python environment, i.e. the
36-
# Python environment corresponding to `which python`. This may be an
37-
# activated virtual environment.
38-
# - "persistent_uv": use a dedicated project-scoped virtual environment that is
39-
# located at <project_root>/`.venv` and is managed by `uv`. Subprocesses are
40-
# launched with `uv run`.
33+
# If this table is defined, it must have only a single key-value pair. See below for options.
4134
#
4235
# Defaults to "active".
43-
python_environment = "active"
36+
[tool.dg.project.python_environment]
37+
38+
# (default): use the currently active Python environment, i.e. the
39+
# Python environment corresponding to `which python`. This may be an
40+
# activated virtual environment.
41+
active = true
42+
43+
# - "uv_managed": use a dedicated project-scoped virtual environment that is
44+
# located at <project_root>/`.venv` and is managed by `uv`. Subprocesses are
45+
# launched with `uv run`. The active virtual environment is ignored
46+
# uv_managed = true
4447

4548
[tool.dg.cli]
4649

examples/docs_snippets/docs_snippets/guides/dg/scaffolding-project/1-scaffolding-project.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@ dg init --project-name my-project
22

33
Creating a Dagster project at /.../my-project.
44
Scaffolded files for Dagster project at /.../my-project.
5-
...
5+
You can create additional projects later by running 'dg scaffold project'.

examples/docs_snippets/docs_snippets/guides/dg/scaffolding-project/2-tree.txt

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,7 @@ tree
1111
│   │   └── __init__.py
1212
│   └── lib
1313
│   └── __init__.py
14-
├── tests
15-
│   └── __init__.py
16-
└── uv.lock
14+
└── tests
15+
└── __init__.py
1716

18-
7 directories, 7 files
17+
7 directories, 6 files

examples/docs_snippets/docs_snippets/guides/dg/workspace/5-scaffold-project.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
dg scaffold project projects/project-2
1+
dg scaffold project projects/project-2 --python-environment uv_managed
22

33
Creating a Dagster project at /.../dagster-workspace/projects/project-2.
44
Scaffolded files for Dagster project at /.../dagster-workspace/projects/project-2.

examples/docs_snippets/docs_snippets_tests/snippet_checks/guides/components/test_component_docs_adding_attributes_to_assets.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ def test_components_docs_adding_attributes_to_assets(
3535
with isolated_snippet_generation_environment() as get_next_snip_number:
3636
# Scaffold code location, add some assets
3737
_run_command(
38-
cmd="dg scaffold project my-project --use-editable-dagster && cd my-project/src",
38+
cmd="dg scaffold project my-project --python-environment uv_managed --use-editable-dagster && cd my-project/src",
3939
)
4040
_run_command(
4141
cmd="dg scaffold dagster.asset team_a/subproject/a.py",

examples/docs_snippets/docs_snippets_tests/snippet_checks/guides/components/test_components_docs.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,9 @@ def next_snip_no():
5050
)
5151

5252
# Scaffold project
53+
# TODO: Make this use "active" python environment in docs followup
5354
run_command_and_snippet_output(
54-
cmd="dg scaffold project jaffle-platform --use-editable-dagster",
55+
cmd="dg scaffold project jaffle-platform --python-environment uv_managed --use-editable-dagster",
5556
snippet_path=COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-scaffold.txt",
5657
update_snippets=update_snippets,
5758
snippet_replace_regex=[

examples/docs_snippets/docs_snippets_tests/snippet_checks/guides/components/test_components_docs_creating_a_component.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ def test_components_docs_index(
3636
with isolated_snippet_generation_environment() as get_next_snip_number:
3737
# Scaffold code location
3838
_run_command(
39-
cmd="dg scaffold project my-component-library --use-editable-dagster && cd my-component-library",
39+
cmd="dg scaffold project my-component-library --python-environment uv_managed --use-editable-dagster && cd my-component-library",
4040
)
4141

4242
#########################################################

0 commit comments

Comments
 (0)