Skip to content

Commit ca061de

Browse files
gibsondanalangenfeld
authored andcommitted
Add an opt-in env var to re-enable emitting partition subsets from planned events in OSS (#29288)
## Summary & Motivation Ths fix in #28113 trades off correctness for performance - failed status on assets is reliably correct now, but single-run backfills may take a lot longer to create runs since they are creating Until we can make the requisite schema changes to give us the best of both worlds (single event per asset per run, and also correct partition failure calculations) make the default to be slower but correct, with the ability oto opt in if you prefer faster but not always correct. ## How I Tested These Changes Ran TestSqliteEventLogStorage with this new env var set to True
1 parent 58ca178 commit ca061de

File tree

1 file changed

+6
-1
lines changed
  • python_modules/dagster/dagster/_core/storage/event_log

1 file changed

+6
-1
lines changed

python_modules/dagster/dagster/_core/storage/event_log/base.py

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
from abc import ABC, abstractmethod
23
from collections.abc import Iterable, Mapping, Sequence
34
from typing import TYPE_CHECKING, AbstractSet, NamedTuple, Optional, Union # noqa: UP035
@@ -40,6 +41,7 @@
4041
from dagster._record import record
4142
from dagster._utils import PrintFn
4243
from dagster._utils.concurrency import ConcurrencyClaimStatus, ConcurrencyKeyInfo
44+
from dagster._utils.tags import get_boolean_tag_value
4345
from dagster._utils.warnings import deprecation_warning
4446

4547
if TYPE_CHECKING:
@@ -563,7 +565,10 @@ def supports_global_concurrency_limits(self) -> bool:
563565

564566
@property
565567
def supports_partition_subset_in_asset_materialization_planned_events(self) -> bool:
566-
return False
568+
# Setting this environment variable will cause a single planned event to be emitted for
569+
# each asset for a run with a single run backfill, instead of one per partition
570+
# (but will also cause partitions to not be marked as failed if the run fails
571+
return get_boolean_tag_value(os.getenv("DAGSTER_EMIT_PARTITION_SUBSET_IN_PLANNED_EVENTS"))
567572

568573
@property
569574
def asset_records_have_last_observation(self) -> bool:

0 commit comments

Comments
 (0)