Releases: dagster-io/dagster
1.11.3 (core) / 0.27.3 (libraries)
New
-
Introduced
AssetExecutionContext.load_asset_value
, which enables loading asset values from the IO manager dynamically rather than requiring asset values be loaded as parameters to the asset function. For example:@dg.asset(deps=[the_asset]) def the_downstream_asset(context: dg.AssetExecutionContext): return context.load_asset_value(dg.AssetKey("the_asset"))
-
Expose asset_selection parameter for
submit_job_execution
function in DagsterGraphQLClient, thanks @brunobbaraujo! -
Large error stack traces from Dagster events will be automatically truncated if the message or stack trace exceeds 500kb. The exact value of the truncation can be overridden by setting the
DAGSTER_EVENT_ERROR_FIELD_SIZE_LIMIT
environment variable. -
Added
databento
,ax
, andbotorch
kind tags, thanks @aleewen and @CompRhys! -
[dagster-k8s] Added the option to include
ownerReferences
s to k8s executor step jobs, ensuring that the step job and step pod are properly garbage collected if the run pod is deleted. These can be enabled by setting theenable_owner_references
flag on the executor config. -
[components] Added
dg list component-tree
command which can be used to visualize the component tree of a project. -
[components] Added the ability to reference, load, and build defs for other components in the same project. In YAML, you may use the
load_component_at_path
andbuild_defs_at_path
functions:type: dagster.PythonScriptComponent attributes: execution: path: my_script.py assets: - key: customers_export deps: - "{{ load_component_at_path('dbt_ingest').asset_key_for_model('customers') }}"
Bugfixes
- [components] Python component instances are now properly loaded from ordinary Python files.
- Fixed an issue that could cause asset backfills to request downstream partitions at the same time as their parent partitions in rare cases.
- Fixed a bug that could cause
@graph_asset
s to not properly apply theAllPartitionMapping
orLastPartitionMapping
to dependencies, thanks @BoLiuV5! - Fixed a bug that could cause code locations to fail to load when a custom python AutomationCondition was used as the operand of
AutomationCondition.any_deps_match()
orAutomationCondition.all_deps_match()
. - The
create-dagster
standalone executable now works on all Linux versions using glibc 2.17 or later. - [ui] Partition tags are now properly shown on the runs page, thanks @HynekBlaha!
- [ui] Using the "Retry from Asset Failure" option when retrying a run that failed after materializing all of its assets will now correctly indicate that there is no work that needs to be retried.
- [ui] The timeline tab on the Overview page now shows runs by sensor when they were launched by an automation condition sensor, instead of showing every row in the same "Automation condition" row.
- [ui] Fixed an issue where filtering to an asset group on the lineage page did not apply the correct repository filter in code locations with multiple repositories.
- [ui] Fixed an issue where asset checks referencing asset keys that did not exist in the asset graph did not appear in the Dagster UI.
- [ui] Fixed occasional crashes of the asset graph on the asset lineage tab.
- [dagster-dbt] The
@dbt_assets
decorator and associated APIs no longer error when parsing dbt projects that contain an owner with multiple emails.
Documentation
- Fixed typos in the ELT pipeline tutorial, thanks @aaronprice00 and @kevJ711!
- Fixed typos in components docs, thanks @tintamarre!
- Fixed error in Sling docs, thanks @nhuray!
- Updated the
AutomationCondition.replace
type signature to provide callers more information about the returnedAutomationCondition
, thanks @dschafer!
Dagster Plus
- Catalog search now uses a similar syntax to the selection syntax for filtering by attribute (eg:
Code location: location
->code_location: location
.
1.11.2 (core) / 0.27.2 (libraries)
New
- The
dagster
package now supports Python 3.13. - [dagster-tableau] Tableau assets can now be subsetted and materialized individually. #31078
- [dagster-snowflake-polars] The new
dagster-snowflake-polars
package adds aSnowflakePolarsIOManager
that can be used to read and write Snowflake tables using Polars.
Bugfixes
- [ui] Fixed some cases where strings would be incorrectly middle-truncated.
Documentation
- [dbt] Fixed incorrect reference to
dbt_profiles.yml
in thedagster-dbt
tutorial (thanks @EFox2413!). - [dbt] Added documentation for the new dbt Cloud integration.
Dagster Plus
- [ui][observe] You can now bulk add/remove assets to/from your favorites.
1.11.1 (core) / 0.27.1 (libraries)
New
dagster definitions validate
will now raise an exception if there are invalid partition mappings between any assets in your asset graph (for example, an upstream and downstream asset with time-based partitions definitions using different timezones).- Performance improvements for run dequeuing when there are many queued runs using pools.
- [ui] For times in the last two days, Dagster UI now shows e.g. "47 hours ago" instead of "2 days ago."
- [ui] Asset checks now show whether they are
blocking
. - [dagster-tableau] Tableau workbooks fetched in Dagster can now be filtered and selected using the WorkbookSelectorFn.
Bugfixes
@graph
now correctly allows omitting inputs when the destinations of an input mapping have a default value.@record
classes no longer create problematic namespace conflicts with the symbol "check."- [ui] Filtering by partition on the Asset Events view now works as expected.
- [ui] Assets without definitions can now be properly wiped in the Asset Catalog.
Documentation
- Added clearer setup instructions for Windows and Linux users to the Contributing documentation, thanks @oohwooh!
- Fixed broken links in the Contributing documentation, thanks @emmanuel-ferdman!
Dagster Plus
- [ui] Fixed an issue that prevented status filtering from working within the selection syntax.
dg & Components
- Fixed an issue where
dg scaffold github-actions
would invoke thedg
CLI with outdated parameters for serverless orgs. - [dagster-dlt] Fixed an issue where the default scaffolded dlt load led to an invalid asset key.
1.11.0 (core) / 0.27.0 (libraries)
Major changes since 1.10.0 (core) / 0.26.0 (libraries)
Components — Configurable, reusable building blocks for data pipelines (now stable)
Components, first released as Preview in 1.10.6, have reached Release Candidate status. APIs are stable, fully documented, and are the recommended, production-ready defaults for new Dagster projects.
- Simplified interface: A short YAML block in
defs.yaml
, or a lightweightComponent
subclass in Python, lets you spin up arbitrary Dagster definitions (such as assets, resources, schedules, checks, and more), removing boilerplate while keeping every definition type-checked. - Custom components: Subclassing
Component
lets you wrap any internal script or third-party tool behind a strongly-typed interface; get the same autocompletion and docs as first-party integrations. - Pythonic templating: Register variables or helpers with
@template_var
so pipeline authors edit parameters directly in YAML without touching Python. Optional inline components keep small bits of Python co-located. - Powerful tooling: High quality errors, strongly-typed schemas, rich CLI support, and auto-generated docs directly in the UI.
- Read the docs: https://docs.dagster.io/guides/build/components/
dg
— the everything-CLI for developer experience (now stable)
The dg
CLI provides a single surface for scaffolding, local iteration, execution, and static analysis; introduced as Preview in 1.10.6, it has also reached Release Candidate status.
- Scaffolding: Namespaced scaffold commands such as
dg scaffold defs dagster.asset assets.py
,dg scaffold component …
to quickly generate definitions without boilerplate. - Local development & ad-hoc execution:
dg dev
spins up local instance with UI;dg launch
runs jobs and assets from the CLI. - Introspection & checks:
dg list
enumerates definitions;dg check
validates YAML and Python code. - Utility bundle:
dg utils
provides support for Cursor/VSCode schema setup, JSON-schema export, and deep component inspection. - CLI reference: https://docs.dagster.io/api/dg/dg-cli
create-dagster
— one-shot project scaffold
create-dagster
scaffolds a ready-to-run Dagster project or workspace in one command (pipx, uvx, brew, curl friendly).
create-dagster project
supersedes thedagster project scaffold
flow with the modernsrc/
+defs/
layout and a pre-wired localdg
CLI, with no active Python environment required.- Docs: https://docs.dagster.io/guides/build/projects/creating-a-new-project
Core Orchestration
- Retry from asset failure with multi‑assets – a new re‑execution option allows rerunning only failed assets in multi-asset steps, rather than all assets within a failed step.
- Checks emitted from ops –
AssetCheckEvaluation
objects can now be yielded from within ops. - Per‑asset hooks –
@asset
now accepts ahooks
argument for success/failure callbacks. - Backfill improvements
BackfillPolicy
is now GA- Backfills can now use a threadpool for more efficient run submission. By default, the daemon will now use 4 workers.
- Concurrency enhancements – run blocking is now on by default for concurrency pools, preventing oversubscription when scheduling runs.
- FreshnessPolicy — A new
FreshnessPolicy
API is introduced, replacing the deprecatedFreshnessPolicy
API (which has been renamed toLegacyFreshnessPolicy
). The API is under active development, and will eventually also supersede freshness checks as the primary way of specifying and evaluating asset freshness. For more details, check out the docs: https://docs.dagster.io/guides/labs/freshness
UI
- Unified asset selection syntax lets you combine lineage traversal, attribute filters, and boolean logic in a single expression; the same grammar powers Alerts, Insights, Saved Selections, the Asset Catalog, and Components YAML. An analogous op-selection grammar is available in the Gantt view of a single run. [docs]
- Redesigned, customizable asset-graph nodes with health overlays and deeper zoom.
- Runs › Backfills consolidates all backfill activity under the Runs page for faster navigation.
Integrations
- Fivetran integration GA: the
FivetranWorkspace
resource is now GA [docs]. - Airflow (Beta): Airflow Component lets you surface Airflow DAGs inside Dagster for mixed-orchestrator observability [docs].
- dbt Cloud (Beta): first-class job launches and lineage capture [docs].
- Apache Iceberg (Preview): Iceberg IOManager writes/reads lake-house tables [docs].
- Integrations Marketplace (Preview): “Integrations” tab to browse first- and third-party integrations natively in Dagster UI (enable via User Settings → “Display integrations marketplace”).
Changes since 1.10.21 (core) / 0.25.21 (libraries)
New
MaterializeResult
now optionally supports avalue
parameter. If set, the asset's IOManager will be invoked. You may also optionally annotate your return types with-> MaterializeResult[T]
to indicate the specific value type you expect.- Allow importing
FreshnessPolicy
fromdagster.deprecated
. - Adds a custom error message when importing
FreshnessPolicy
from thedagster
module. freshness_policy
parameter now used to pass the new freshness policies (InternalFreshnessPolicy
) to asset specs, asset decorator, etc.- Removed
@preview
from@definitions
. - [components] Introduce
build_defs_for_component
, which can be used to build defs from a component instance outside of adefs
folder. - [components] Removed
@preview
fromDefsFolderComponent
. - [components] Removed
@preview
decorator fromload_from_defs_folder
and enhanced its documentation with detailed usage instructions and examples. - [components] The
asset_post_processors
field onSlingReplicationCollectionComponent
andAirflowInstanceComponent
is no longer supported, and has been replaced with the top-levelpost_processors
field. - [dagster-tableau] Tableau workbooks fetched in Dagster can now be filtered and selected using the WorkbookSelectorFn.
- [dagster-dbt]
dagster-dbt
now supports dbt-core 1.10. - [dagster-dbt] dbt tests with error severity are now modeled as blocking asset checks, ensuring that if a run fails due to a dbt test failure, the connected model is included in a retried run if it is retried using the experimental "Enable retries from asset failure" feature. This change should not result in any behavior changes during execution since the dbt cli already fails the step and any downstream models if dbt tests fail with error severity, but could change the behavior that depends on blocking tests.
- [dagster-sigma] When fetching data from the sigma API, the
SigmaOrganization
resource will now use an exponential backoff strategy in response to getting rate limited instead of immediately failing. - [dagster-sling] Removed
asset_post_processors
onSlingReplicationCollectionComponent
and uses genericpost_processing
key at top-level instead. - [dagster-pandera] Adds support for version 0.24.0 of the
pandera
library todagster-pandera
, dropping support for pandera 0.23.1 and below. - [ui] Show whether an asset check is
blocking
.
Bugfixes
- [dagster-dlt] Fixed an issue where the default scaffolded dlt load led to an invalid asset key.
- Fixed a bug with
DAGSTER_GRPC_SENSOR_TIMEOUT_SECONDS
not being propagated through from daemon to code servers, resulting in the sensor still timing out at 60 seconds if thedagster code-server start
entrypoint was used. - [dagster-sling] Fixed an issue with the
SlingResource
that could cause values specified withEnvVar
s to provide the env var name instead of the env var value to the sling replication configuration. - [ui] Fix timestamps on the "Recent events" view on some assets.
- [ui] Fix "View" link color on code location status toasts.
Breaking Changes
Definitions
andAssetsDefinition
will now error if they get differentAssetSpec
s with the same key.- Renamed
FreshnessPolicy
toLegacyFreshnessPolicy
.
Deprecations
- [dagster-sling] The
SlingReplicationCollectionComponent
is now configured by passingconnections
directly. This means that thesling
yaml field and theresource
python argument are both deprecated, and will be removed in a future release. Theconnections
field in yaml now shares a format with Sling'senv.yaml
. - [components] The
load_defs
entrypoint has been deprecated in favor ofload_from_defs_folder
, which takes a single path for a folder to load definitions from rather than a module object. - [components] The no longer recommended
inline-component
subcommand ofdg scaffold defs
is now hidden from --help. - [components]
load_defs
is no longer public.
Dagster Plus
- The billing page has been updated to show available plans front-and-center and recommend a plan based on trial usage.
- The trial "days remaining" banner and "choose a plan" CTAs have been re-designed.
1.10.21 (core) / 0.26.21 (libraries)
New
- [dagster-tableau] The
tableau_assets
decorator is now available to create the asset definitions of all materializable assets in a given Tableau workspace. These assets can be subsetted and materialized using theTableauCloudWorkspace.refresh_and_poll
method. - [dagster-tableau] The deprecated
get_*_asset_key
methods on theDagsterTableauTranslator
class have been removed. - [ui] Show tags for a Job on the sidebar of the Job page. #30728
Bugfixes
- Fixed a bug where "falsey" defualt config values were not showing up in the launchpad. This has been fixed for all cases except the empty dictionary.
- Fixed an issue with the new "re-execute from asset failure" functionality that could cause additional steps to be included if the job was previously re-executed from step failure.
- Fixed an issue where the
staleStatusByPartition
,staleCausesByPartition
, anddataVersionByPartition
fields on the graphqlAssetNode
would fail when called on an asset with dynamic partitions. - [dagster-dbt] Fixed an issue where creating a DagsterDbtTranslator that didn't call the parent class's constructor in its
__init__
method would raise an Exception during execution. - [dagster-sling] Removed upper-bound pin on the
sling
dependency. - [dagster-sling] Fixed an issue with the
SlingResource
that could cause values specified withEnvVar
s to provide the env var name instead of the env var value to the sling replication configuration. - [dagster-fivetran] Introduced a
dagster-fivetran snapshot
command, allowing Fivetran workspaces to be captured to a file for faster subsequent loading.
Dagster Plus
- [ui] The integrations marketplace (currently behind a feature flag) now allows you to create, edit, and delete private packages.
dg & Components (Preview)
- A clean and informative error message is now printed when an invalid set of parameters is passed to
dg scaffold defs ...
. - Component asset specs now allow specifying partition definitions through the
partitions_def
key. - The
dg
cache is no longer operative, sincedg
now operates in the same python environment as the projects it manipulates. Config options for the cache have been removed. - The
load_defs
entrypoint has been deprecated in favor ofload_from_defs_folder
, which takes a single path for a folder to load definitions from rather than a module object. - The
asset_post_processors
field onSlingReplicationCollectionComponent
andAirflowInstanceComponent
is no longer supported, and has been replaced with the top-levelpost_processors
field. - Fixed an issue where dg projects using
autoload_defs=true
could not be deployed to Dagster+.
1.10.20 (core) / 0.26.20 (libraries)
New
- The
@asset
decorator now supports ahooks
argument to allow for functions to be executed on asset success / failure (thanks @brunobbaraujo)! - The log message produced when an asset check is evaluated now includes its pass / fail state.
- The
@multi_asset_check
decorator now supports thepool
argument. - [dagster-dbt] The
DagsterDbtTranslator
class now has aget_asset_check_spec
method which can be overridden to customize theAssetCheckSpecs
that are produced for each individual dbt test.
Bugfixes
- Fixed an issue where setting the non-public
blocking
attribute on anAssetCheckSpec
would halt execution of the step as soon as any asset check failure was emitted, even if the step still had asset materializations or check evaluations to emit that were not downstream of the failed asset check. Now that this issue has been fixed, theblocking
attribute onAssetCheckSpec
has been made a public attribute. If you were making use of theblocking
attribute on AssetCheckSpec before it was public and relying on the previous behavior, you should exit from your asset evaluation function after emitting an AssetCheckFailure from within your multi-asset in order to halt further execution of the step. - Fixed a bug where
GraphDefinition.to_job()
would not work if an op had a custom IO manager key. - Fixed an issue that would cause
.allow()
and.ignore()
applications to not propagate through.since()
automation conditions. - Fixed a bug where assets with cross-location dependencies could sometimes be incorrectly reported as "Unsynced".
- Fixed an issue where
dagster dev
was sometimes failing to load code locations with a "Deadline Exceeded" error unless the--use-legacy-code-server-behavior
flag was set. - The backfill daemon can now be configured to use a threadpool executor via helm (thanks @HynekBlaha)!
- [dagster-gcp] Added a
google-cloud-bigquery>=1.28.3
pin to correctly reflect the lowest compatible version.
Breaking Changes
- [ui] Moved legacy Auto-materialize (global AMP) tab from Overview to Automations.
Dagster Plus
- Fixed an issue where certain rare network conditions could cause steps to hang while uploading compute logs after a step finished.
- [ui] For users with the new Observe UIs enabled, the Asset Health and Resources tabs are no longer shown on the Timeline page.
dg & Components (Preview)
- Fix a bug with
dg check yaml
where valid component type names were rejected if they were not registered (i.e. visible fromdg check components
). dg create-dagster
now warns when scaffolding a project or workspace if it is not the latest version.- The
project.registry_modules
configuration can now accept wildcards (e.g.foo_bar.components.*
). This will register any module matching the pattern withdg
. - The
env
YAML function now errors if the specified env var is unset. Default values can be provided as an additional argument:{{ env('MY_ENV_VAR', 'default') }}
- defs.yaml files can now specify a component in the module where it is defined, as opposed to just the module where it is exposed in the
dg
registry. - The
PipesSubprocessScriptCollectionComponent
has been removed. - Running dg commands such as
dg check defs
anddg dev
in a project folder that is part of the workspace will now only apply to that project, instead of every project in the workspace. - Scaffolded projects no longer contain a "components" directory or a Python
dagster_dg_cli.plugin
entry point. - Scaffolded components can now be placed anywhere within a project module hierarchy.
- The entry point group used by shared libraries exposing custom components to
dg
has been renamed fromdagster_dg_cli.plugin
todagster_dg_cli.registry_modules
(projects no longer need to define an entry point group at all). dg list plugin-modules
has been renamed todg list registry-modules
.dg list defs
now supports configuring output columns with the--columns/-c
option.- [dagster-airbyte] Introduced a
AirbyteCloudWorkspaceComponent
which can be used to pull in Airbyte Cloud connections into Dagster
1.10.19 (core) / 0.26.19 (libraries)
New
- The database table used by the
DbIOManager
is now configurable via"table"
output (asset) metadata key #30310 - Changed default settings for backfill daemon to
use_threads=True
,num_workers=4
. Thanks @HynekBlaha! - A new function
build_asset_check_context
can be used to build asset check contexts for direct invocation. - Changed
Definitions.get_all_asset_specs
to only return assets directly passed in as AssetSpecs or AssetsDefinitions. - Removed
selection
argument fromDefinitions.map_asset_specs
. If selection is needed use the newDefinitions.map_resolved_asset_specs
. Definitions.get_job_def
now warns when it finds an unresolved job or no job.- Changed
Definitions.get_assets_def
to return an AssetsDefinition without resolving if it was passed in directly. - [dagster-dbt]
build_schedule_from_dbt_selection
now supports aselector
argument, allowing you to use yaml-based selectors. - [dagster-k8s] Pods created by the Kubernetes run launcher and executor from Dagster Plus now include the
dagster/deployment-name
label. - [dagster-pipes] Pipes execution errors are no longer treated as framework errors, meaning they properly invoke RetryPolicies.
- [helm] Backfill daemon configuration now supported. Thanks @HynekBlaha!
- [ui] Show relative start time on runs in run timeline hover lists. #30327
Bugfixes
- [ui] Fixed live updating of asset materialization statuses in asset graph.
dg & Components (Preview)
- Running dg commands like
dg check defs
anddg dev
in a project folder that is part of the workspace will now only apply to that project, instead of every project in the workspace. dg list defs
now supports the--assets/-a
option, to show only assets matching the provided selection.dg list defs
now supports a--path
argument to subset the defs files shown.- The
create-dagster workspace
command now accepts the same required path argument as thecreate-dagster project
command, instead of defaulting to adagster-workspace
subfolder of the current working directory. - The entry point group used by shared libraries exposing custom components to dg has been renamed from
dagster_dg_cli.plugin
todagster_dg_cli.registry_modules
(projects no longer need to define an entry point group at all). dg list plugin-modules
has been renamed todg list registry-modules
.- Newly scaffolded projects no longer contain a "components" directory or a Python
dagster_dg_cli.plugin
entry point. - Newly scaffolded components can now be placed anywhere within a project module hierarchy.
Resolvable
subclasses can now use baredict
andlist
as field types.- Resolving a
Resolvable
subclass will no longer change empty strings toNone
. - Users can define multiple
@definitions
-decorated functions in a single module in thedefs
hierarchy and they are automatically merged and incorporated into the project. - Added
@component_instance
to replace@component
. This allows multiple component instances in a python file. - Fixed an issue where
dg
commands would sometimes output extradagster_telemetry_logger
lines to stdout at the end of commands. - Added
@template_var
as an alternative approach for defining variables in a templating context.
1.10.18 (core) / 0.26.18 (libraries)
New
BackfillPolicy
is now marked as generally available (GA).- Optimized the order of the
@asset
decorator overloads to make custom wrappers around the asset decorator easier. Thanks @jonathanunderwood! - [dagster-slack] Added
get_client()
to SlackResource.
Bugfixes
Definitions
andAssetDefinition
will now warn if they get differentAssetSpec
s with the same key. This will become an exception in 1.11.- Functions that load all definitions from modules like
load_definitions_from_modules
now handle duplicateAssetSpec
s. - Fixed typo in logging. Thanks @eli-b!
- [dagster-dbt] An issue occurring when using dbt selection arguments with a dbt project using saved queries and semantic models has been fixed.
- [dagster-fivetran] Fixed an issue with
load_assets_from_fivetran_instance
where assets whose asset keys have been customized using a Fivetran translator would lead to an exception.
Documentation
- Fixed grammar issues in GCP docs. Thanks @D1n0!
- Fixed missing docs for
required_resource_keys
in@sensor
. Thanks @seyf97!
Breaking Changes
Definitions
andAssetDefinition
will now warn if they get differentAssetSpec
s with the same key. This will become an exception in 1.11.
Dagster Plus
- [dagster-cloud-cli] Added a
--wait
to thedagster-cloud job launch
command that makes it wait until the launched run copmletes. Thanks @stevenayers! - [fix][dagster-cloud-cli] Fixed an issue where the
dagster-cloud
cli failed to deploy PEX projects on python 3.12 in certain environments without setuptools already installed.
dg & Components (Preview)
- The
dg
CLI is now installed in each project's Python environment instead of as a global tool. If you previously haddg
installed globally and are upgrading, first uninstall the global dg tool (uv tool uninstall dagster-dg
orpip uninstall dagster-dg
) and adddagster-dg-cli
as a dependency to each of your projects. Newly scaffolded projects will automatically include thedg
CLI going forward. - A new
create-dagster
CLI has been added for scaffolding projects instead of workspaces.uvx create-dagster project
has replaceddg scaffold project
, anduvx create-dagster workspace
has replaceddg scaffold workspace
. The commands take identical arguments. - Definitions and component instances are now scaffolded with
dg scaffold defs <scaffolder>
instead ofdg scaffold <scaffolder>
. - The
component.yaml
file to specify a component instance is now called "defs.yaml". "component.yaml" will remain supported for several weeks, but is deprecated. - The
lib
folder in newly scaffolded projects has been renamed to "components". dg scaffold component-type
has been renamed todg scaffold component
.dg list plugins
has been renamed todg list plugin-modules
and now outputs only plugin module names, not plugin objects.dg list component
now lists component types instead of instances.- Exports from
dagster.components
are now available in the top-leveldagster
module. - Added
@component_instance
to replace@component
, which allows multiple component instances in a python file. - If you type a partial component name for
dg scaffold defs <component>
, you will now be prompted with possible matches. - When component classnames are unique, you can now use just the classname as an alias for the fully qualified name when running
dg scaffold defs <component>
. dg launch
now supports passing config files through--config/-c
.- Added
Component.from_attributes_dict
andComponent.from_yaml_path
to help with testing. - Added
dagster.components.testing.component_defs
utility to help with testing components. - Scaffolded schedules and sensors in
dg
are now loadable by default. dg scaffold defs inline-component
can now be used to create inline components and corresponding instances.dg list defs
now outputs resources.- [fix] The
dagster-dbt
cli flag--components
flag now correctly findsDbtProjectComponent
again. - [dagster-fivetran] Added a
FivetranAccountComponent
which can be used to pull in Fivetran connections into Dagster. - [dagster-dlt] The DltLoadCollectionComponent scaffolder no longer attempts to automatically construct loads for the given source and destination type.
- Fixed an issue where components failed to load when using
load_assets_from_airbyte_instance
or other APIs that return aCacheableAssetsDefinition
.
1.10.17 (core) / 0.26.17 (libraries)
Bugfixes
- Fixed an issue where an error was displayed in the UI while viewing run logs
- [dagster-dbt] Fixed an issue occurring when using dbt selection arguments with a dbt project using semantic models.
1.10.16 (core) / 0.26.16 (libraries)
New
typing_extensions
is now pinned to>=4.11.0
instead of>=4.10.0
.- [ui] Viewing an automation condition evaluation now automatically expands the set of applicable sub-conditions.
- [ui] Added the ability to navigate from an automation condition evaluation to upstream automation condition evaluations.
- [ui] Added an asset graph node facet for viewing automation conditions and the most recent evaluation. This can be enabled in the user settings via a feature flag (
Enable faceted asset nodes
). - [ui] A new experimental integrations marketplace tab is now available and can be enabled in your user settings via a feature flag (
Display integrations marketplace
). It provides easy access to the gallery of dagster-supported plugins.
Bugfixes
- Fixed an issue with the
ExternalNotebookDataRequest
GRPC call that would allow it to access files outside of the current directory. - Fixed an issue that would cause
op_tags
set on@observable_source_asset
s to be dropped from the underlying step context object when executed. - Fixed an issue where
dagster dev
would sometimes raise a gRPC error when loading several code locations at once. - Fixed an issue where setting an environment variable to the string "false", "0" or "None" for a dagster config field using a
BoolSource
would evaluate to True. - Fixed an issue where specifying
executable_path
in a workspace.yaml file to run code locations in a different virtual environment would not correctly inherit the PATH of that virtual environment in the code location. - [dagster-dbt] Fixed an issue causing dbt CLI invocation to fail when materializing assets when
OpExecutionContext
was used as the type hint for the context. - [dagster-deltalake] Corrected the
timeout
property data type inClientConfig
to be str instead of int (thanks, @edsoncezar16!)
Documentation
- Added additional config fields to the
K8sRunLauncher
example (thanks, @nishan-soni!) - Corrected broken links on the automation landing page (thanks, @briandailey!)
Dagster Plus
- [ui] Alert policy event tags no longer appear red and yellow outside of the policy notification history.
dg & Components (Preview)
- Added suggestions to component model error messages when using built-in models for common classes such as
AssetKey
andAssetSpec
. dg list env
now displays whether env vars are configured in each Dagster Plus scope.- Introduced
Resolver.passthrough()
to avoid processing fields on a component model. ResolvedAssetKey
is now exported fromdagster.components
.dg init
has been removed.dg scaffold project
anddg scaffold workspace
should be used instead.- Fixed an issue where
dg dev
failed with a temporarily file permissions error when running on Windows. Thanks @polivbr!