Releases: dagster-io/dagster
Releases · dagster-io/dagster
1.10.15 (core) / 0.26.15 (libraries)
New
- Added a config section to
dagster.yaml
to enable submitting backfill runs in a threadpool. - Expanded definition time validation for partition mappings to avoid runtime errors querying asset status.
- [ui][beta] You can now re-execute a run that targeted a multi-asset from the point of asset failure instead of step failure, meaning only assets that failed or were skipped will be re-executed. To enable this option, turn on the
Enable retries from asset failure
feature flag in your user settings. - [ui] Made it easier to select and copy image names for code locations.
- [ui] Added asset lineage navigation within the automation condition evaluation tree.
- [ui] Viewing an evaluation tick now auto-expands the set of applicable automation conditions.
- [ui] Added an asset graph node facet for viewing automation conditions and the most recent evaluation in the global asset graph.
- [dagster-fivetran] The
FivetranWorkspace
resource is now marked as generally available (GA).
Bugfixes
- Changed asset wipes to also wipe associated asset check evaluations.
- [dagster-fivetran] Fixed an issue causing the Fivetran integration to fail when the schema config does not exist for a connector.
Documentation
- Fixed a broken link in the airflow migration docs. Thanks @jjyeo!
- Updated example snippet to include a missing type hint. Thanks @agrueneberg!
Deprecations
- [dagster-fivetran] The
FivetranResource
resource is now deprecated. Use the newFivetranWorkspace
resource instead.
dg & Components (Preview)
- Changed
Scaffolder.scaffold
to have the params object as an attribute of theScaffoldRequest
object instead of a dictionary. This is a breaking change for those who have implemented a custom scaffolder. - Added support for scaffolding resources via
dg scaffold dagster.resources path/to/resources.py
. - Added support for the usage of
@definitions
in thedefs
hierarchy. - Dagster components now include code references by default. When viewing an asset emitted by a component in the asset catalog, this will allow you to jump to the backing
component.yaml
file in your editor. - [dagster-dbt]
DbtProjectComponent
fields now properly evaluate templates. - [dagster-sling] Updated the SlingReplicationCollectionComponent from using the
asset_attributes
parameter totranslation
, in order to match our other integration components. - Fixed an issue where
dg dev
failed with a temporary file permissions error when running on Windows. Thanks @polivbr!
1.10.14 (core) / 0.26.14 (libraries)
New
- [dagster-tableau] Refined Tableau integration for API 3.25 or greater.
- [dagster-tableau] Data sources with extracts can now be materialized in Tableau assets created with
build_tableau_materializable_assets_definition
. - [ui] Added kinds tag for treasuredata.
- [ui] Add Supabase kind icon.
Bugfixes
- Fixed a bug which could cause an error when calling
MultiPartitionsDefinition.has_partition_key()
on invalid keys. - Fixed a bug where the default multiprocess executor would fail runs where the child process for a step crashed, even if a retry policy resulted in a successful retry of that crashed step.
- Fixed a bug with
AutomationCondition.initial_evaluation
which could cause it to returnFalse
for an asset that went from having a condition, to having no condition at all, back to having the original condition again. - [ui] Fixed an issue which could cause the "Target" field of AutomationConditionSensorDefinitions to render incorrectly when exactly one asset check was defined in a code location.
- [dagster-dbt] Fix
DagsterDbtTranslatorSettings.enable_source_tests_as_checks
returning duplicate asset checks.
Documentation
- Added a sample Dagster+ ECS CloudFormation template which incorporates private subnets.
- Fixed incorrect storage values in the Fargate task section of the AWS deployment guide, thanks @alexpotv!
- Updated log stream docs, thanks @jjyeo!
- Fixed broken code in the configurable resources guide, thanks @nightscape!
Deprecations
dagster.InitResourceContext.dagster_run
has been deprecated in favor ofInitResourceContext.run
.
dg & Components (Preview)
- [dagster-k8s] PipesK8sComponent has been added.
- Dagster components no longer change the working directory while they are being loaded. This allows components to store relative paths and ensure that they will still work when accessed outside of the component loading codepath. This change may affect user-defined components that depend on
Path.cwd()
oros.getcwd()
. Instead, you should use a path relative to the current source file when loading paths in a component, using thecontext.resolve_source_relative_path
method (seeresolve_dbt_project
inDbtProjectComponent
for an example). - Added
dagster.job
scaffolder. - [dagster-dbt] The
DbtProjectComponent
now has atranslation_settings
argument for adjustingDagsterDbtTranslatorSettings
. - [dagster-dbt][fix]
DbtProjectComponent
fields now properly evaluate templates. - Fixed docstring for
load_defs
entrypoint, thanks @mattgiles!
1.10.13 (core) / 0.26.13 (libraries)
New
- If an unselected asset check is executed during a run, the system will now warn instead of throwing a hard error.
- When evaluating
AutomationCondition.any_deps_match
orAutomationCondition.all_dep_match
with an allow / ignore specified, an error will no longer be produced if the provided asset selection references an asset key that does not exist. - Added the ability to restrict the list of ports that
dagster dev
is allowed to use to open subprocesses when running on Windows, by setting theDAGSTER_PORT_RANGE
env var to a string of the form<start>=<end>
- for example "20000-30000". - [dagster-aws] The S3 sensor's
get_objects
now returns an empty list if no new files can be found since thesince_last_modified
parameter. Thanks @bartcode! - [dagster-dbt]
@dbt_assets
andbuild_dbt_manifest_asset_selection
now support aselector
argument, allowing you to use yaml-based selectors. - [dagster-dbt] you can now enable dbt source tests as asset checks by using the flag
DagsterDbtTranslatorSettings.enable_source_tests_as_checks
. - [dagster-k8s] improved run monitoring when running with increased backoff limits. Thanks @adam-bloom!
Bugfixes
- Fixed a bug with
AutomationCondition.initial_evaluation
which could cause it to returnFalse
for an asset that went from having a condition, to having no condition at all, back to having the original condition again. - Fixed a bug that would cause the
AutomationCondition.any_deps_updated()
condition to evaluate toFalse
when evaluated on a self-dependency. - Fixed a bug in the
quickstart_aws
example. Thanks @Thenkei! - [ui] Fixed navigation between asset tabs by no longer preserving query parameters from one tab to the next.
- [ui] Fixed an issue where the asset graph looked like it was still loading when it wasn't.
Documentation
- Added Scala Spark / Dagster Pipes guide.
dg & Components (Preview)
- [dg] Error message when an invalid configuration file is detected is now shorter and more clear.
- [components] Descriptions and examples have been restored for core models such as
ResolvedAssetSpec
.description
andexamples
arguments have been added toResolver
for documenting fields on non-pydantic model basedResolvable
classes.
1.10.12 (core) / 0.26.12 (libraries)
New
- [ui] Removed the
By partition
grouping view for recent events for assets that do not have a definition in the workspace. - [ui] The asset graph now displays asset health information when the new Observe UI feature flag is enabled.
- [ui] A new feature flag allows you to customize the appearance of assets on the asset graph by enabling and disabling individual facets.
- [ui] Fixed a bug that prevented filtering asset events by type.
- [dagster-k8s] K8sPipeClient failures will now include the last 100 lines of the logs of the pod that failed, instead of the full log output.
Bugfixes
- Fixed an issue which caused multi assets that were automatically broken apart in some contexts to remain separated even in cases where this was not necessary to maintain execution dependencies.
- Fixed a bug that would cause multi-assets defined with
can_subset=True
to error when usingdagster-pipes
if not all outputs were emitted. - [ui] Fixed an issue where the asset graph looked like it was still loading when it wasn't.
Documentation
dg & Components (Preview)
dg
will now fail with an error message if it's version is below the minimum supported version for the version ofdagster
in your environment.dg
now checks for new versions and prompts the user to update. The check runs automatically atdg
startup once per day.dg
will now emit a warning prompting the user to reinstall the package if it detects an entry point in project metadata that does not show up when running indg list plugins
.- Added
dagster.multi_asset
scaffolder - Added
dagster.asset_check
scaffolder - Fixed a bug where
dg list defs
would crash if anything was written to stdout while loading a project's definitions. - The default location for the
dg
user config file on Unix has been moved from~/.dg.toml
to~/.config/dg.toml
.~/.config
can be overridden by setting$XDG_CONFIG_HOME
. - Cache deserialization errors are now ignored. Previously, when the
dg
contents of the cache were in an outdated format,dg
could crash. Now it will just rebuild the cache. - The Components ETL Pipeline Tutorial now supports users of both
pip
anduv
. - The
dg
CLI will now emit a warning if you are using "active" mode for your project python environment, there is a virtual environment at<project_root>/.venv
, and the activated venv is not<project_root>/.venv
- A new
dg
settingcli.suppress_warnings
is available. This takes a list of warning types to suppress. - Added a warning message to inform users they need to install their project package when skipping automatic environment setup.
- Changed configuration of project Python environments. The
tool.dg.project.python_environment
previously accepted a string,"active"
or"persistent_uv"
. Now it accepts a table with one of two keys:{active = true}
: equivalent of previous"active"
{uv_managed = true}
: equivalent of previous"persistent_uv"
- Changed the default python environment for newly scaffolded projects to
tool.dg.project.python_environment
to{active = true}
. This means by default, no virtual environment oruv.lock
will be created when scaffolding a new project (viadg init
ordg scaffold project
). You can pass--python-environment uv_managed
for the old behavior. - Removed the
--skip-venv
flag ondg scaffold project
anddg init
. - The
dagster_components
package has been merged intodagster
and use of thedagster-components
package has been deprecated.dagster-components
will remain as a stub package for the next few weeks, but code should be updated to import fromdagster.components
instead ofdagster_components
. - [dagster-dbt] the
dagster-dbt project prepare-and-package
cli now supports--components
for handlingDbtProjectComponent
- [dagster-dbt]
DbtProjectComponent
has been reworked, changing both the python api and the yaml schema.dbt
has been replaced withproject
with a slightly different schema, andasset_attributes
withtranslation
.
1.10.11 (core) / 0.26.11 (libraries)
New
- [ui] Runs launched from the Dagster UI get a
dagster/from_ui = true
tag, making it easy to filter for them. - [ui] The run page now shows the number of log levels selected as well as the number of log levels available.
- Added
AirflowFilter
API for use withdagster-airlift
, allows you to filter down the set of dags retrieved up front for perf improvements.
Bugfixes
- [ui] Fixed a bug that prevented filtering asset events by type.
- Fixed a bug that would cause multi-assets defined with
can_subset=True
to error when usingdagster-pipes
if not all outputs were emitted. - [dagster-dbt] the
state_path
argument toDbtCliResource
now resolves relative to the project directory as documented. - [dagster-k8s] Made reliability improvements to PipesK8sClient log streaming when transient networking errors occur. The default behavior of the PipesK8sClient is now to reconnect to the stream of logs every hour (this value can be overridden by setting the
DAGSTER_PIPES_K8S_CONSUME_POD_LOGS_REQUEST_TIMEOUT
environment variable) and to retry up to 5 times if an error occurs while streaming logs from the launched Kubernetes pod (this value can be overridden by setting theDAGSTER_PIPES_K8S_CONSUME_POD_LOGS_RETRIES
environment variable.) - [dagster-dbt] Fixed a bug where dbt jobs would fail due to unparseable logs causing errors in
DbtCliInvocation.stream_raw_events
. (Thanks @ross-whatnot!)
Dagster Plus
- [ui] It is now possible to roll back a code location to a version that had previously been in an errored state.
dg & Components (Preview)
- The
dg
CLI will now emit a warning if you are using "active" mode for your project python environment, there is a virtual environment at<project_root>/.venv
, and the activated venv is not<project_root>/.venv
- A new
dg
settingcli.suppress_warnings
is now available. This takes a list of warning types to suppress. - Changed configuration of project Python environments. The
tool.dg.project.python_environment
previously accepted a string,"active"
or"persistent_uv"
. Now it accepts a table with one of three keys: -{active = true}
: equivalent of previous"active"
-{uv_managed = true}
: equivalent of previous"persistent_uv"
- Changed the default python environment for newly scaffolded projects to
tool.dg.project.python_environment
to{active = true}
. This means by default, no virtual environment oruv.lock
will be created when scaffolding a new project (viadg init
ordg scaffold project
). You can pass--python-environment uv_managed
for the old behavior. - Removed the
--skip-venv
flag ondg scaffold project
anddg init
. - Fixed a bug where new projects scaffolded with
dg scaffold project
were lackingdagster
as a dependency. - The "Creating a library of components" guide has been replaced by a new and more general "Creating a
dg
plugin" guide. - The arguments/options of the
dg init
command have changed. You may pass.
as an argument to initialize a project/workspace in the CWD. Seedg init --help
for more details. - The
dagster-components
package (which was converted to a stub package in the last release) is no longer being published. Alldagster-components
functionality is now part ofdagster
. - Projects should now expose custom component types under the
dagster_dg.plugin
entry point group instead ofdagster_dg.library
.dagster_dg.library
support is being kept for now for backcompatibility, but will be dropped in a few weeks. - Fixed formatting of line added to
project/lib/__init__.py
when scaffolding a component type. - The
dg env list
command is nowdg list env
- The
dg plus env pull
command is nowdg plus pull env
. - The
dg list component-type
command has been removed. There is a newdg list plugins
with output that is a superset ofdg list component-type
. - The
dg
user config file on Unix is now looked for at~/.dg.toml
instead of~/dg.toml
. - [dagster-dbt]
DbtProjectComponent
has been reworked, changing both the python api and the yaml schema.dbt
has been replaced withproject
with a slightly different schema, andasset_attributes
withtranslation
.
1.10.10 (core) / 0.26.10 (libraries)
New
- A new
blocking
parameter has been added tobuild_last_update_freshness_checks
andbuild_time_partition_freshness_checks
. - The default byte size limit for gRPC requests and responses is now 100MB instead of 50MB. This value can be adjusted by setting the
DAGSTER_GRPC_MAX_RX_BYTES
andDAGSTER_GRPC_MAX_SEND_BYTES
environment variables on the gRPC client and server processes. - Added a new
Definitions.map_asset_specs
method, which allows for the transformation of properties on any AssetSpec or AssetsDefinition objects in theDefinitions
object which match a given asset selection. Definitions.validate_loadable
anddagster definitions validate
will now raise an error on assets with invalid partition mappings, like aTimeWindowPartitionMapping
between two time-based partitions definitions with different timezones. Previously, these invalid partition mappings would not raise an error until they were used to launch a run.- [dagster-k8s] Reliability improvements to
PipesK8sClient
log streaming when transient networking errors occur. The default behavior of thePipesK8sClient
is now to reconnect to the stream of logs every 3600 seconds (this value can be overridden by setting theDAGSTER_PIPES_K8S_CONSUME_POD_LOGS_REQUEST_TIMEOUT
environment variable) and to retry up to 5 times if an error occurs while streaming logs from the launched Kubernetes pod (this value can be overridden by setting theDAGSTER_PIPES_K8S_CONSUME_POD_LOGS_RETRIES
environment variable.)
Bugfixes
- Fixed an issue where run monitoring sometimes didn't fail runs that were stuck in a
NOT_STARTED
status instead of aSTARTING
status. - Fixed an issue where Dagster run metrics produced large amounts of error lines when running in containers without a CPU limit set on the container.
- Fixed an issue where using partial resources in Pydantic >= 2.5.0 could result in an unexpected keyword argument TypeError. (Thanks @HynekBlaha!)
- Fixed an issue with new
AutomationCondition.executed_with_tags()
that would cause thetag_keys
argument to not be respected. Updated the display name ofAutomationCondition.executed_with_tags()
to contain the relevant tag_keys and tag_values. - [ui] Fixed a bug preventing filtering on the asset events page.
- [ui] Fix custom time datepicker filter selection for users with custom Dagster timezone settings.
- [dagster-fivetran] Fixed a bug causing the Fivetran integration to fetch only 100 connectors per destination.
- [dagster-fivetran] Paused connectors no longer raise an exception in
FivetranWorkspace.sync_and_poll(...)
. Instead, they skip and a warning message is logged. - [dagster-fivetran] Fixed an issue where new runs of code locations using Fivetran assets would sometimes raise a "Failure condition: No metadata found for CacheableAssetsDefinition" error if the run was started immediately after a new version of the code location was deployed.
- [dagster-cloud] Reliability improvements to the
dagster-cloud job launch
command when launching runs targeting large numbers of assets or asset checks.
Dagster Plus
Fixed an issue with the identification of the base repository URL the DAGSTER_CLOUD_GIT_URL
. The raw URL is now exposed as DAGSTER_CLOUD_RAW_GIT_URL
.
dg & Components (Preview)
- The
dagster_components
package has been merged intodagster
. All symbols exported fromdagster_components
are now exported bydagster
. - Projects should now expose custom component types under the
dagster_dg.plugin
entry point group instead ofdagster_dg.library
.dagster_dg.library
support is being kept for now for backcompatibility, but will be dropped in a few weeks. Component.get_schema
has been renamed toComponent.get_model_cls
. Override that instead to customize the frontend of your component.Component.get_schema
will continue to work for the time being but will be removed at some point in the future.- Add support for
dg.toml
files.dg
settings inpyproject.toml
are set undertool.dg
, but indg.toml
they are set at the top level. Adg.toml
may be used in place ofpyproject.toml
at either the project or workspace level. dg
-scaffolded workspaces now include adg.toml
instead ofpyproject.toml
file.- Projects scaffolded using
dg init
ordg scaffold project
now follow modern Python packaging conventions, placing the root module in a top-levelsrc
directory and usehatchling
as build backend rather thansetuptools
. - Scaffolding a component type defaults to inheriting from
dagster.components.Model
instead of getting decorated with@dataclasses.dataclass
. - Assets scaffolded using
dg scaffold dagster.asset
will no longer be commented out. - The
dg list component-type
command has been removed. There is a newdg list plugins
with output that is a superset ofdg list component-type
. dg list defs
now includes infomation about asset checks.- Fix formatting of line added to
project/lib/__init__.py
when scaffolding a component type. - Fixed bug where schedules were displayed in the sensors section of
dg list defs
. - Fixed a bug where
dg
would crash when working with Python packages with an src-based layout.
1.10.9 (core) / 0.26.9 (libraries)
Bugfixes
- [ui] Fix custom time datepicker filter selection for users with custom Dagster timezone settings.
dg & Components (Preview)
- Add support for
dg.toml
files.dg
settings inpyproject.toml
are set undertool.dg
, but indg.toml
they are set at the top level. Adg.toml
may be used in place ofpyproject.toml
at either the project or workspace level. dg
-scaffolded workspaces now include adg.toml
instead ofpyproject.toml
file.dg
-scaffolded projects now place the root package in ansrc/
directory and usehatchling
as the build backend rather thansetuptools
.- Fixed a bug where
dg
would crash when working with Python packages with an src-based layout. dg check yaml
now properly validates component files in nested subdirectories of thedefs/
folder.
1.10.8 (core) / 0.26.8 (libraries)
New
- [ui] The Dagster UI now allows you to specify extra tags when re-executing runs from failure from the runs feed re-execute dialog, or by holding shift when clicking Re-execute menu items throughout the app.
- [ui] Performance improvements for loading the partitions page for multi-partitioned assets.
- [ui] Fix link in toast messages that appear when launching backfills.
- [ui] Dagster's UI now allows you to copy run tags as a YAML block from the Tags and Configuration modals.
- [ui] The Dagster Run UI now allows you to view the execution plan of a queued run.
Bugfixes
- The
AutomationCondition.initial_evaluation
condition has been updated to become true for all partitions of an asset whenever the PartitionsDefinition of that asset changes, rather than whenever the structure of the condition changes. - [dagster-fivetran] Fixed an issue where new runs of code locations using fivetran assets would sometimes raise a "Failure condition: No metadata found for CacheableAssetsDefinition" error if the run was started immediately after a new version of the code location was deployed.
- [dagster-fivetran] Fixed an issue where including multiple sets of assets from
build_fivetran_assets_definitions
in a singleDefinitions
object would result in "duplicate node" errors when launching a run. - [ui] Fixed line charts for colorblind themes.
- [ui] Fixed an issue with querystring parsing that can arise when selecting a large number of items in the selection syntax input.
- [ui] Fixed tag filtering on automations list.
- [ui] Fixed hover state on focused inputs.
- [ui] Fixed an issue with the Run step selection input autocomplete where it would suggest
key:"*substring*"
instead ofname:"*substring*"
. - [ui] Fixed the "View run" link shown when launching runs
Documentation
- Fix a bug in example code for pyspark.
dg & Components (Preview)
- Added the ability to scaffold Python components.
- The
DefsModuleComponent
has been renamed toDefsFolderComponent
. - When scaffolding a component, the command is now
dg scaffold my_project.ComponentType
instead ofdg scaffold component my_project.ComponentType
. - [dagster-dg]
dagster list defs
will now read environment variables from a local .env file if present when constructing the definitions. dagster-components
has been merged intodagster
and use of thedagster-components
package has been deprecated.
dagster-components
will remain as a stub package for the next few weeks, but code should be updated to import fromdagster.components
instead ofdagster_components
.- The
DbtProjectComponent
has been relocated to thedagster-dbt
package,
importable asdagster_dbt.DbtProjectComponent
. - The
SlingReplicationCollectionComponent
has been relocated to thedagster-sling
package,
importable asdagster_sling.SlingReplicationCollectionComponent
.
1.10.7 (core) / 0.26.7 (libraries)
New
- Applying changes from sensor test results now also applies changes from dynamic partition requests.
- When merging assets from multiple code locations, autogenerated specs are now prioritized lower than customized external asset specs.
- [ui] Allowed using command-click to view a run from the toast message that appears when starting a materialization of an asset.
- [ui] Asset graph can now zoom out a lot more.
- [ui] Added a kind tag for dbt Cloud.
- [dagster-dlt] Added backfill policy to dlt_assets, defaulting to single-run. (Thanks @neuromantik33!)
Bugfixes
- Updated
AutomationCondition.initial_evaluation
condition to become true for all partitions of an asset whenever the PartitionsDefinition of that asset changes, rather than whenever the structure of the condition changes. - Fixed a bug with several integrations that caused data fetched from external APIs not to be properly cached during code server initialization, leading to unnecessary API calls in run and step worker processes. This affected
dagster-airbyte
,dagster-dlift
,dagster-dbt
,dagster-fivetran
,dagster-looker
,dagster-powerbi
,dagster-sigma
, anddagster-tableau
. - [ui] Fixed an issue with the Run step selection input autocomplete where it would suggest
key:"*substring*"
instead ofname:"*substring*"
. - [ui] Fixed the "View run" link shown when launching runs.
- [ui] Fixed an issue where updating a catalog view caused an infinite loading state.
- Fixed an issue which could cause asset check evaluations emitted from the body of the op to not impact the check status of an asset in the UI.
- Fixed an issue that could cause an asset backfill created by re-executing another backfill from the point of failure to error on the first tick in rare cases.
- Fixed an issue that could cause automation condition evaluations to fail to render in the UI in rare cases.
- [ui] Fixed a regression in the "Cancel Backfill" option for job backfills that have finished queuing runs.
- [ui] Fixed overflow of long runs feed table on backfill page.
- [dagster-dbt] Replaced
@validator
with@field_validator
in dagster_dbt/core/resource.py to prevent Pydantic deprecation warnings. (Thanks @tintamarre!)
Documentation
- Updated the "Asset versioning and caching" guide to reflect the current Dagster UI and "Unsynced" labels.
- Removed a few repeated lines in documentation on customizing automation conditions. (Thanks @zero-stroke!)
- Fixed example in TableRecord documentation to use the new input format.
Configuration
- [dagster-gcp] Updated Dataproc configuration to the latest version. If necessary, consider pinning your
dagster-gcp
version while you migrate config. Please see the full list of changed fields: https://gist.github.com/deepyaman/b4d562e04fe571e40037a344b7a9937d - [dagster-aws][dagster-spark] Updated Spark configuration to the latest version (3.5.5). If necessary, consider pinning your
dagster-aws
and/ordagster-spark
version while you migrate config. Please see the full list of changed fields: https://gist.github.com/deepyaman/f358f5a70fea28d5f164aca8da3dee04
Dagster Plus
- [ui] Fixed filtering for multiple tags on list view pages, including Automations.
- [ui] Fixed an issue where the urls generated by catalog filtering would remove all filters if loaded directly.
- [ui] Added a warning on the sign-in page indicating that the sign-in and signup flows will be changing soon.
- [ui] Require confirmation when rolling back a code location to a previous version.
dg & Components (Preview)
- Virtual environment detection settings for projects have changed. Previously, the global settings
use_dg_managed_environment
andrequire_local_venv
controlled the environment used when launching project subprocesses. This is now configured at the project level. Thetool.dg.project.python_environment
setting takes a value of eitherpersistent_uv
oractive
.persistent_uv
will be used by default in new scaffolded projects and uses a uv-managed.venv
in the project root.active
is the default if notool.dg.project.python_environment
is set, and just uses the active python environment and opts out ofdg
management of the environment. - A new base class,
Resolvable
, has been added. This can be used to simplify the process of defining a yaml schema for your components. Instead of manually defining a manualResolvedFrom[...]
andResolvableModel
, the framework will automatically derive the model schema for you based off of the annotations of your class. - Python files with Pythonic Components (i.e. defined with
@component
) can now contain relative imports. - The
dg init
command now accepts optional--workspace-name
and--project-name
options to allow scaffolding an initial workspace and project via CLI options instead of prompts. - Added a new
dagster_components.dagster.DefsFolderComponent
that can be used at any level of yourdefs/
folder to apply asset attributes to the definitions at or below that level. This was previously nameddagster_components.dagster.DefsModuleComponent
.
1.10.6 (core) / 0.26.6 (libraries)
New
- Added a new
AutomationCondition.executed_with_tags()
condition that makes it possible to filter for updates from runs with particular tags. AssetCheckEvaluation
can now be yielded from Dagster ops to log an evaluation of an asset check outside of an asset context.- Added the
kinds
argument todagster.AssetOut
, allowing kinds to be specified in@multi_asset
. - [dagster-dbt]
AssetCheckEvaluations
are now yielded fromops
leveragingDbtCliResource.cli(...)
when asset checks are included in the dbt asset lineage. - [dagster-sling] The
SlingResource.replicate()
method now takes an optionstream
parameter, which allows events to be streamed as the command executes, instead of waiting until it completes (thanks, @natpatchara-w!). - [dagster-graphql] The
DagsterGraphQLClient
now supports anauth
keyword argument, which is passed to the underlyingRequestsHTTPTransport
constructor. - [ui] The asset selection syntax input now allows slashes "/" in the freeform search.
- [ui] The backfill pages now show summary information on all tabs for easier backfill monitoring.
Bugfixes
- Fixed issue with
AutomationCondition.newly_requested()
which could cause it to fail when nested withinAutomationCondition.any_deps_match()
orAutomationCondition.all_deps_match()
. - Fixed a bug with
AutomationCondition.replace()
that would cause it to not effectAutomationCondition.since()
conditions. - Fixed a bug with several integrations that caused data fetched from external APIs not to be properly cached during code server initialization, leading to unnecessary API calls in run and step worker processes. This affected:
dagster-airbyte
,dagster-dlift
,dagster-dbt
,dagster-fivetran
,dagster-looker
,dagster-powerbi
,dagster-sigma
, anddagster-tableau
. - Fixed a bug that could cause invalid circular dependency errors when using asset checks with additional dependencies.
- [dagster-fivetran] Loading assets for a Fivetran workspace containing incomplete and broken connectors now no longer raises an exception.
- [ui] Fixed the colorblind (no red/green) theme behavior when in dark mode.
- [ui] The Asset > Partitions page no longer displays an error in some cases when creating dynamic partitions.
- [ui] The Launch and Report Events buttons no longer error if you click it immediately after creating a new dynamic partition.
dg & Components (Preview)
__pycache__
files are no longer included in the output ofdg list component
(thanks @stevenayers!)- When resolving the
deps
of anAssetSpec
from yaml, multi-part asset keys are now correctly parsed (thanks @stevenayers!) - The entrypoint group for dg projects has been renamed from
dagster.components
todagster_dg.library
dg check yaml
is now run by default beforedg dev
anddg check defs