Skip to content

Releases: dagster-io/dagster

1.11.3 (core) / 0.27.3 (libraries)

24 Jul 20:19
Compare
Choose a tag to compare

New

  • Introduced AssetExecutionContext.load_asset_value, which enables loading asset values from the IO manager dynamically rather than requiring asset values be loaded as parameters to the asset function. For example:

    @dg.asset(deps=[the_asset])
    def the_downstream_asset(context: dg.AssetExecutionContext):
      return context.load_asset_value(dg.AssetKey("the_asset"))
  • Expose asset_selection parameter for submit_job_execution function in DagsterGraphQLClient, thanks @brunobbaraujo!

  • Large error stack traces from Dagster events will be automatically truncated if the message or stack trace exceeds 500kb. The exact value of the truncation can be overridden by setting the DAGSTER_EVENT_ERROR_FIELD_SIZE_LIMIT environment variable.

  • Added databento, ax, and botorch kind tags, thanks @aleewen and @CompRhys!

  • [dagster-k8s] Added the option to include ownerReferencess to k8s executor step jobs, ensuring that the step job and step pod are properly garbage collected if the run pod is deleted. These can be enabled by setting the enable_owner_references flag on the executor config.

  • [components] Added dg list component-tree command which can be used to visualize the component tree of a project.

  • [components] Added the ability to reference, load, and build defs for other components in the same project. In YAML, you may use the load_component_at_path and build_defs_at_path functions:

    type: dagster.PythonScriptComponent
    
    attributes:
      execution:
        path: my_script.py
      assets:
        - key: customers_export
          deps:
            - "{{ load_component_at_path('dbt_ingest').asset_key_for_model('customers') }}"

Bugfixes

  • [components] Python component instances are now properly loaded from ordinary Python files.
  • Fixed an issue that could cause asset backfills to request downstream partitions at the same time as their parent partitions in rare cases.
  • Fixed a bug that could cause @graph_assets to not properly apply the AllPartitionMapping or LastPartitionMapping to dependencies, thanks @BoLiuV5!
  • Fixed a bug that could cause code locations to fail to load when a custom python AutomationCondition was used as the operand of AutomationCondition.any_deps_match() or AutomationCondition.all_deps_match().
  • The create-dagster standalone executable now works on all Linux versions using glibc 2.17 or later.
  • [ui] Partition tags are now properly shown on the runs page, thanks @HynekBlaha!
  • [ui] Using the "Retry from Asset Failure" option when retrying a run that failed after materializing all of its assets will now correctly indicate that there is no work that needs to be retried.
  • [ui] The timeline tab on the Overview page now shows runs by sensor when they were launched by an automation condition sensor, instead of showing every row in the same "Automation condition" row.
  • [ui] Fixed an issue where filtering to an asset group on the lineage page did not apply the correct repository filter in code locations with multiple repositories.
  • [ui] Fixed an issue where asset checks referencing asset keys that did not exist in the asset graph did not appear in the Dagster UI.
  • [ui] Fixed occasional crashes of the asset graph on the asset lineage tab.
  • [dagster-dbt] The @dbt_assets decorator and associated APIs no longer error when parsing dbt projects that contain an owner with multiple emails.

Documentation

  • Fixed typos in the ELT pipeline tutorial, thanks @aaronprice00 and @kevJ711!
  • Fixed typos in components docs, thanks @tintamarre!
  • Fixed error in Sling docs, thanks @nhuray!
  • Updated the AutomationCondition.replace type signature to provide callers more information about the returned AutomationCondition, thanks @dschafer!

Dagster Plus

  • Catalog search now uses a similar syntax to the selection syntax for filtering by attribute (eg: Code location: location -> code_location: location.

1.11.2 (core) / 0.27.2 (libraries)

10 Jul 21:08
Compare
Choose a tag to compare

New

  • The dagster package now supports Python 3.13.
  • [dagster-tableau] Tableau assets can now be subsetted and materialized individually. #31078
  • [dagster-snowflake-polars] The new dagster-snowflake-polars package adds a SnowflakePolarsIOManager that can be used to read and write Snowflake tables using Polars.

Bugfixes

  • [ui] Fixed some cases where strings would be incorrectly middle-truncated.

Documentation

  • [dbt] Fixed incorrect reference to dbt_profiles.yml in the dagster-dbt tutorial (thanks @EFox2413!).
  • [dbt] Added documentation for the new dbt Cloud integration.

Dagster Plus

  • [ui][observe] You can now bulk add/remove assets to/from your favorites.

1.11.1 (core) / 0.27.1 (libraries)

03 Jul 18:35
Compare
Choose a tag to compare

New

  • dagster definitions validate will now raise an exception if there are invalid partition mappings between any assets in your asset graph (for example, an upstream and downstream asset with time-based partitions definitions using different timezones).
  • Performance improvements for run dequeuing when there are many queued runs using pools.
  • [ui] For times in the last two days, Dagster UI now shows e.g. "47 hours ago" instead of "2 days ago."
  • [ui] Asset checks now show whether they are blocking.
  • [dagster-tableau] Tableau workbooks fetched in Dagster can now be filtered and selected using the WorkbookSelectorFn.

Bugfixes

  • @graph now correctly allows omitting inputs when the destinations of an input mapping have a default value.
  • @record classes no longer create problematic namespace conflicts with the symbol "check."
  • [ui] Filtering by partition on the Asset Events view now works as expected.
  • [ui] Assets without definitions can now be properly wiped in the Asset Catalog.

Documentation

  • Added clearer setup instructions for Windows and Linux users to the Contributing documentation, thanks @oohwooh!
  • Fixed broken links in the Contributing documentation, thanks @emmanuel-ferdman!

Dagster Plus

  • [ui] Fixed an issue that prevented status filtering from working within the selection syntax.

dg & Components

  • Fixed an issue where dg scaffold github-actions would invoke the dg CLI with outdated parameters for serverless orgs.
  • [dagster-dlt] Fixed an issue where the default scaffolded dlt load led to an invalid asset key.

1.11.0 (core) / 0.27.0 (libraries)

26 Jun 19:42
Compare
Choose a tag to compare

Major changes since 1.10.0 (core) / 0.26.0 (libraries)

Components — Configurable, reusable building blocks for data pipelines (now stable)

Components, first released as Preview in 1.10.6, have reached Release Candidate status. APIs are stable, fully documented, and are the recommended, production-ready defaults for new Dagster projects.

  • Simplified interface: A short YAML block in defs.yaml, or a lightweight Component subclass in Python, lets you spin up arbitrary Dagster definitions (such as assets, resources, schedules, checks, and more), removing boilerplate while keeping every definition type-checked.
  • Custom components: Subclassing Component lets you wrap any internal script or third-party tool behind a strongly-typed interface; get the same autocompletion and docs as first-party integrations.
  • Pythonic templating: Register variables or helpers with @template_var so pipeline authors edit parameters directly in YAML without touching Python. Optional inline components keep small bits of Python co-located.
  • Powerful tooling: High quality errors, strongly-typed schemas, rich CLI support, and auto-generated docs directly in the UI.
  • Read the docs: https://docs.dagster.io/guides/build/components/

dgthe everything-CLI for developer experience (now stable)

The dg CLI provides a single surface for scaffolding, local iteration, execution, and static analysis; introduced as Preview in 1.10.6, it has also reached Release Candidate status.

  • Scaffolding: Namespaced scaffold commands such as dg scaffold defs dagster.asset assets.py, dg scaffold component … to quickly generate definitions without boilerplate.
  • Local development & ad-hoc execution: dg dev spins up local instance with UI; dg launch runs jobs and assets from the CLI.
  • Introspection & checks: dg list enumerates definitions; dg check validates YAML and Python code.
  • Utility bundle: dg utils provides support for Cursor/VSCode schema setup, JSON-schema export, and deep component inspection.
  • CLI reference: https://docs.dagster.io/api/dg/dg-cli

create-dagster — one-shot project scaffold

create-dagster scaffolds a ready-to-run Dagster project or workspace in one command (pipx, uvx, brew, curl friendly).

Core Orchestration

  • Retry from asset failure with multi‑assets – a new re‑execution option allows rerunning only failed assets in multi-asset steps, rather than all assets within a failed step.
  • Checks emitted from ops  – AssetCheckEvaluation objects can now be yielded from within ops.
  • Per‑asset hooks – @asset now accepts a hooks argument for success/failure callbacks.
  • Backfill improvements
    • BackfillPolicy is now GA
    • Backfills can now use a threadpool for more efficient run submission. By default, the daemon will now use 4 workers.
  • Concurrency enhancements – run blocking is now on by default for concurrency pools, preventing oversubscription when scheduling runs.
  • FreshnessPolicy — A new FreshnessPolicy API is introduced, replacing the deprecated FreshnessPolicy API (which has been renamed to LegacyFreshnessPolicy). The API is under active development, and will eventually also supersede freshness checks as the primary way of specifying and evaluating asset freshness. For more details, check out the docs: https://docs.dagster.io/guides/labs/freshness

UI

  • Unified asset selection syntax lets you combine lineage traversal, attribute filters, and boolean logic in a single expression; the same grammar powers Alerts, Insights, Saved Selections, the Asset Catalog, and Components YAML. An analogous op-selection grammar is available in the Gantt view of a single run. [docs]
  • Redesigned, customizable asset-graph nodes with health overlays and deeper zoom.
  • Runs › Backfills consolidates all backfill activity under the Runs page for faster navigation.

Integrations

  • Fivetran integration GA: the FivetranWorkspace resource is now GA [docs].
  • Airflow (Beta): Airflow Component lets you surface Airflow DAGs inside Dagster for mixed-orchestrator observability [docs].
  • dbt Cloud (Beta): first-class job launches and lineage capture [docs].
  • Apache Iceberg (Preview): Iceberg IOManager writes/reads lake-house tables [docs].
  • Integrations Marketplace (Preview): “Integrations” tab to browse first- and third-party integrations natively in Dagster UI (enable via User Settings → “Display integrations marketplace”).

Changes since 1.10.21 (core) / 0.25.21 (libraries)

New

  • MaterializeResult now optionally supports a value parameter. If set, the asset's IOManager will be invoked. You may also optionally annotate your return types with -> MaterializeResult[T] to indicate the specific value type you expect.
  • Allow importing FreshnessPolicy from dagster.deprecated.
  • Adds a custom error message when importing FreshnessPolicy from the dagster module.
  • freshness_policy parameter now used to pass the new freshness policies (InternalFreshnessPolicy) to asset specs, asset decorator, etc.
  • Removed @preview from @definitions.
  • [components] Introduce build_defs_for_component, which can be used to build defs from a component instance outside of a defs folder.
  • [components] Removed @preview from DefsFolderComponent.
  • [components] Removed @preview decorator from load_from_defs_folder and enhanced its documentation with detailed usage instructions and examples.
  • [components] The asset_post_processors field on SlingReplicationCollectionComponent and AirflowInstanceComponent is no longer supported, and has been replaced with the top-level post_processors field.
  • [dagster-tableau] Tableau workbooks fetched in Dagster can now be filtered and selected using the WorkbookSelectorFn.
  • [dagster-dbt] dagster-dbt now supports dbt-core 1.10.
  • [dagster-dbt] dbt tests with error severity are now modeled as blocking asset checks, ensuring that if a run fails due to a dbt test failure, the connected model is included in a retried run if it is retried using the experimental "Enable retries from asset failure" feature. This change should not result in any behavior changes during execution since the dbt cli already fails the step and any downstream models if dbt tests fail with error severity, but could change the behavior that depends on blocking tests.
  • [dagster-sigma] When fetching data from the sigma API, the SigmaOrganization resource will now use an exponential backoff strategy in response to getting rate limited instead of immediately failing.
  • [dagster-sling] Removed asset_post_processors on SlingReplicationCollectionComponent and uses generic post_processing key at top-level instead.
  • [dagster-pandera] Adds support for version 0.24.0 of the pandera library to dagster-pandera, dropping support for pandera 0.23.1 and below.
  • [ui] Show whether an asset check is blocking.

Bugfixes

  • [dagster-dlt] Fixed an issue where the default scaffolded dlt load led to an invalid asset key.
  • Fixed a bug with DAGSTER_GRPC_SENSOR_TIMEOUT_SECONDS not being propagated through from daemon to code servers, resulting in the sensor still timing out at 60 seconds if the dagster code-server start entrypoint was used.
  • [dagster-sling] Fixed an issue with the SlingResource that could cause values specified with EnvVars to provide the env var name instead of the env var value to the sling replication configuration.
  • [ui] Fix timestamps on the "Recent events" view on some assets.
  • [ui] Fix "View" link color on code location status toasts.

Breaking Changes

  • Definitions and AssetsDefinition will now error if they get different AssetSpecs with the same key.
  • Renamed FreshnessPolicy to LegacyFreshnessPolicy.

Deprecations

  • [dagster-sling] The SlingReplicationCollectionComponent is now configured by passing connections directly. This means that the sling yaml field and the resource python argument are both deprecated, and will be removed in a future release. The connections field in yaml now shares a format with Sling's env.yaml.
  • [components] The load_defs entrypoint has been deprecated in favor of load_from_defs_folder, which takes a single path for a folder to load definitions from rather than a module object.
  • [components] The no longer recommended inline-component subcommand of dg scaffold defs is now hidden from --help.
  • [components] load_defs is no longer public.

Dagster Plus

  • The billing page has been updated to show available plans front-and-center and recommend a plan based on trial usage.
  • The trial "days remaining" banner and "choose a plan" CTAs have been re-designed.

1.10.21 (core) / 0.26.21 (libraries)

20 Jun 17:43
Compare
Choose a tag to compare

New

  • [dagster-tableau] The tableau_assets decorator is now available to create the asset definitions of all materializable assets in a given Tableau workspace. These assets can be subsetted and materialized using the TableauCloudWorkspace.refresh_and_poll method.
  • [dagster-tableau] The deprecated get_*_asset_key methods on the DagsterTableauTranslator class have been removed.
  • [ui] Show tags for a Job on the sidebar of the Job page. #30728

Bugfixes

  • Fixed a bug where "falsey" defualt config values were not showing up in the launchpad. This has been fixed for all cases except the empty dictionary.
  • Fixed an issue with the new "re-execute from asset failure" functionality that could cause additional steps to be included if the job was previously re-executed from step failure.
  • Fixed an issue where the staleStatusByPartition, staleCausesByPartition, and dataVersionByPartition fields on the graphql AssetNode would fail when called on an asset with dynamic partitions.
  • [dagster-dbt] Fixed an issue where creating a DagsterDbtTranslator that didn't call the parent class's constructor in its __init__ method would raise an Exception during execution.
  • [dagster-sling] Removed upper-bound pin on the sling dependency.
  • [dagster-sling] Fixed an issue with the SlingResource that could cause values specified with EnvVars to provide the env var name instead of the env var value to the sling replication configuration.
  • [dagster-fivetran] Introduced a dagster-fivetran snapshot command, allowing Fivetran workspaces to be captured to a file for faster subsequent loading.

Dagster Plus

  • [ui] The integrations marketplace (currently behind a feature flag) now allows you to create, edit, and delete private packages.

dg & Components (Preview)

  • A clean and informative error message is now printed when an invalid set of parameters is passed to dg scaffold defs ....
  • Component asset specs now allow specifying partition definitions through the partitions_def key.
  • The dg cache is no longer operative, since dg now operates in the same python environment as the projects it manipulates. Config options for the cache have been removed.
  • The load_defs entrypoint has been deprecated in favor of load_from_defs_folder, which takes a single path for a folder to load definitions from rather than a module object.
  • The asset_post_processors field on SlingReplicationCollectionComponent and AirflowInstanceComponent is no longer supported, and has been replaced with the top-level post_processors field.
  • Fixed an issue where dg projects using autoload_defs=true could not be deployed to Dagster+.

1.10.20 (core) / 0.26.20 (libraries)

13 Jun 20:06
Compare
Choose a tag to compare

New

  • The @asset decorator now supports a hooks argument to allow for functions to be executed on asset success / failure (thanks @brunobbaraujo)!
  • The log message produced when an asset check is evaluated now includes its pass / fail state.
  • The @multi_asset_check decorator now supports the pool argument.
  • [dagster-dbt] The DagsterDbtTranslator class now has a get_asset_check_spec method which can be overridden to customize the AssetCheckSpecs that are produced for each individual dbt test.

Bugfixes

  • Fixed an issue where setting the non-public blocking attribute on an AssetCheckSpec would halt execution of the step as soon as any asset check failure was emitted, even if the step still had asset materializations or check evaluations to emit that were not downstream of the failed asset check. Now that this issue has been fixed, the blocking attribute on AssetCheckSpec has been made a public attribute. If you were making use of the blocking attribute on AssetCheckSpec before it was public and relying on the previous behavior, you should exit from your asset evaluation function after emitting an AssetCheckFailure from within your multi-asset in order to halt further execution of the step.
  • Fixed a bug where GraphDefinition.to_job() would not work if an op had a custom IO manager key.
  • Fixed an issue that would cause .allow() and .ignore() applications to not propagate through .since() automation conditions.
  • Fixed a bug where assets with cross-location dependencies could sometimes be incorrectly reported as "Unsynced".
  • Fixed an issue where dagster dev was sometimes failing to load code locations with a "Deadline Exceeded" error unless the --use-legacy-code-server-behavior flag was set.
  • The backfill daemon can now be configured to use a threadpool executor via helm (thanks @HynekBlaha)!
  • [dagster-gcp] Added a google-cloud-bigquery>=1.28.3 pin to correctly reflect the lowest compatible version.

Breaking Changes

  • [ui] Moved legacy Auto-materialize (global AMP) tab from Overview to Automations.

Dagster Plus

  • Fixed an issue where certain rare network conditions could cause steps to hang while uploading compute logs after a step finished.
  • [ui] For users with the new Observe UIs enabled, the Asset Health and Resources tabs are no longer shown on the Timeline page.

dg & Components (Preview)

  • Fix a bug with dg check yaml where valid component type names were rejected if they were not registered (i.e. visible from dg check components).
  • dg create-dagster now warns when scaffolding a project or workspace if it is not the latest version.
  • The project.registry_modules configuration can now accept wildcards (e.g. foo_bar.components.*). This will register any module matching the pattern with dg.
  • The env YAML function now errors if the specified env var is unset. Default values can be provided as an additional argument: {{ env('MY_ENV_VAR', 'default') }}
  • defs.yaml files can now specify a component in the module where it is defined, as opposed to just the module where it is exposed in the dg registry.
  • The PipesSubprocessScriptCollectionComponent has been removed.
  • Running dg commands such as dg check defs and dg dev in a project folder that is part of the workspace will now only apply to that project, instead of every project in the workspace.
  • Scaffolded projects no longer contain a "components" directory or a Python dagster_dg_cli.plugin entry point.
  • Scaffolded components can now be placed anywhere within a project module hierarchy.
  • The entry point group used by shared libraries exposing custom components to dg has been renamed from dagster_dg_cli.plugin to dagster_dg_cli.registry_modules (projects no longer need to define an entry point group at all).
  • dg list plugin-modules has been renamed to dg list registry-modules.
  • dg list defs now supports configuring output columns with the --columns/-c option.
  • [dagster-airbyte] Introduced a AirbyteCloudWorkspaceComponent which can be used to pull in Airbyte Cloud connections into Dagster

1.10.19 (core) / 0.26.19 (libraries)

05 Jun 18:00
Compare
Choose a tag to compare

New

  • The database table used by the DbIOManager is now configurable via "table" output (asset) metadata key #30310
  • Changed default settings for backfill daemon to use_threads=True, num_workers=4. Thanks @HynekBlaha!
  • A new function build_asset_check_context can be used to build asset check contexts for direct invocation.
  • Changed Definitions.get_all_asset_specs to only return assets directly passed in as AssetSpecs or AssetsDefinitions.
  • Removed selection argument from Definitions.map_asset_specs. If selection is needed use the new Definitions.map_resolved_asset_specs.
  • Definitions.get_job_def now warns when it finds an unresolved job or no job.
  • Changed Definitions.get_assets_def to return an AssetsDefinition without resolving if it was passed in directly.
  • [dagster-dbt] build_schedule_from_dbt_selection now supports a selector argument, allowing you to use yaml-based selectors.
  • [dagster-k8s] Pods created by the Kubernetes run launcher and executor from Dagster Plus now include the dagster/deployment-name label.
  • [dagster-pipes] Pipes execution errors are no longer treated as framework errors, meaning they properly invoke RetryPolicies.
  • [helm] Backfill daemon configuration now supported. Thanks @HynekBlaha!
  • [ui] Show relative start time on runs in run timeline hover lists. #30327

Bugfixes

  • [ui] Fixed live updating of asset materialization statuses in asset graph.

dg & Components (Preview)

  • Running dg commands like dg check defs and dg dev in a project folder that is part of the workspace will now only apply to that project, instead of every project in the workspace.
  • dg list defs now supports the --assets/-a option, to show only assets matching the provided selection.
  • dg list defs now supports a --path argument to subset the defs files shown.
  • The create-dagster workspace command now accepts the same required path argument as the create-dagster project command, instead of defaulting to a dagster-workspace subfolder of the current working directory.
  • The entry point group used by shared libraries exposing custom components to dg has been renamed from dagster_dg_cli.plugin to dagster_dg_cli.registry_modules (projects no longer need to define an entry point group at all).
  • dg list plugin-modules has been renamed to dg list registry-modules.
  • Newly scaffolded projects no longer contain a "components" directory or a Python dagster_dg_cli.plugin entry point.
  • Newly scaffolded components can now be placed anywhere within a project module hierarchy.
  • Resolvable subclasses can now use bare dict and list as field types.
  • Resolving a Resolvable subclass will no longer change empty strings to None.
  • Users can define multiple @definitions-decorated functions in a single module in the defs hierarchy and they are automatically merged and incorporated into the project.
  • Added @component_instance to replace @component. This allows multiple component instances in a python file.
  • Fixed an issue where dg commands would sometimes output extra dagster_telemetry_logger lines to stdout at the end of commands.
  • Added @template_var as an alternative approach for defining variables in a templating context.

1.10.18 (core) / 0.26.18 (libraries)

29 May 22:05
Compare
Choose a tag to compare

New

  • BackfillPolicy is now marked as generally available (GA).
  • Optimized the order of the @asset decorator overloads to make custom wrappers around the asset decorator easier. Thanks @jonathanunderwood!
  • [dagster-slack] Added get_client() to SlackResource.

Bugfixes

  • Definitions and AssetDefinition will now warn if they get different AssetSpecs with the same key. This will become an exception in 1.11.
  • Functions that load all definitions from modules like load_definitions_from_modules now handle duplicate AssetSpecs.
  • Fixed typo in logging. Thanks @eli-b!
  • [dagster-dbt] An issue occurring when using dbt selection arguments with a dbt project using saved queries and semantic models has been fixed.
  • [dagster-fivetran] Fixed an issue with load_assets_from_fivetran_instance where assets whose asset keys have been customized using a Fivetran translator would lead to an exception.

Documentation

  • Fixed grammar issues in GCP docs. Thanks @D1n0!
  • Fixed missing docs for required_resource_keys in @sensor. Thanks @seyf97!

Breaking Changes

  • Definitions and AssetDefinition will now warn if they get different AssetSpecs with the same key. This will become an exception in 1.11.

Dagster Plus

  • [dagster-cloud-cli] Added a --wait to the dagster-cloud job launch command that makes it wait until the launched run copmletes. Thanks @stevenayers!
  • [fix][dagster-cloud-cli] Fixed an issue where the dagster-cloud cli failed to deploy PEX projects on python 3.12 in certain environments without setuptools already installed.

dg & Components (Preview)

  • The dg CLI is now installed in each project's Python environment instead of as a global tool. If you previously had dg installed globally and are upgrading, first uninstall the global dg tool (uv tool uninstall dagster-dg or pip uninstall dagster-dg) and add dagster-dg-cli as a dependency to each of your projects. Newly scaffolded projects will automatically include the dg CLI going forward.
  • A new create-dagster CLI has been added for scaffolding projects instead of workspaces. uvx create-dagster project has replaced dg scaffold project, and uvx create-dagster workspace has replaced dg scaffold workspace. The commands take identical arguments.
  • Definitions and component instances are now scaffolded with dg scaffold defs <scaffolder> instead of dg scaffold <scaffolder>.
  • The component.yaml file to specify a component instance is now called "defs.yaml". "component.yaml" will remain supported for several weeks, but is deprecated.
  • The lib folder in newly scaffolded projects has been renamed to "components".
  • dg scaffold component-type has been renamed to dg scaffold component.
  • dg list plugins has been renamed to dg list plugin-modules and now outputs only plugin module names, not plugin objects.
  • dg list component now lists component types instead of instances.
  • Exports from dagster.components are now available in the top-level dagster module.
  • Added @component_instance to replace @component, which allows multiple component instances in a python file.
  • If you type a partial component name for dg scaffold defs <component>, you will now be prompted with possible matches.
  • When component classnames are unique, you can now use just the classname as an alias for the fully qualified name when running dg scaffold defs <component>.
  • dg launch now supports passing config files through --config/-c.
  • Added Component.from_attributes_dict and Component.from_yaml_path to help with testing.
  • Added dagster.components.testing.component_defs utility to help with testing components.
  • Scaffolded schedules and sensors in dg are now loadable by default.
  • dg scaffold defs inline-component can now be used to create inline components and corresponding instances.
  • dg list defs now outputs resources.
  • [fix] The dagster-dbt cli flag --components flag now correctly finds DbtProjectComponent again.
  • [dagster-fivetran] Added a FivetranAccountComponent which can be used to pull in Fivetran connections into Dagster.
  • [dagster-dlt] The DltLoadCollectionComponent scaffolder no longer attempts to automatically construct loads for the given source and destination type.
  • Fixed an issue where components failed to load when using load_assets_from_airbyte_instance or other APIs that return a CacheableAssetsDefinition.

1.10.17 (core) / 0.26.17 (libraries)

23 May 17:45
Compare
Choose a tag to compare

Bugfixes

  • Fixed an issue where an error was displayed in the UI while viewing run logs
  • [dagster-dbt] Fixed an issue occurring when using dbt selection arguments with a dbt project using semantic models.

1.10.16 (core) / 0.26.16 (libraries)

22 May 20:27
Compare
Choose a tag to compare

New

  • typing_extensions is now pinned to >=4.11.0 instead of >=4.10.0.
  • [ui] Viewing an automation condition evaluation now automatically expands the set of applicable sub-conditions.
  • [ui] Added the ability to navigate from an automation condition evaluation to upstream automation condition evaluations.
  • [ui] Added an asset graph node facet for viewing automation conditions and the most recent evaluation. This can be enabled in the user settings via a feature flag (Enable faceted asset nodes).
  • [ui] A new experimental integrations marketplace tab is now available and can be enabled in your user settings via a feature flag (Display integrations marketplace). It provides easy access to the gallery of dagster-supported plugins.

Bugfixes

  • Fixed an issue with the ExternalNotebookDataRequest GRPC call that would allow it to access files outside of the current directory.
  • Fixed an issue that would cause op_tags set on @observable_source_assets to be dropped from the underlying step context object when executed.
  • Fixed an issue where dagster dev would sometimes raise a gRPC error when loading several code locations at once.
  • Fixed an issue where setting an environment variable to the string "false", "0" or "None" for a dagster config field using a BoolSource would evaluate to True.
  • Fixed an issue where specifying executable_path in a workspace.yaml file to run code locations in a different virtual environment would not correctly inherit the PATH of that virtual environment in the code location.
  • [dagster-dbt] Fixed an issue causing dbt CLI invocation to fail when materializing assets when OpExecutionContext was used as the type hint for the context.
  • [dagster-deltalake] Corrected the timeout property data type in ClientConfig to be str instead of int (thanks, @edsoncezar16!)

Documentation

  • Added additional config fields to the K8sRunLauncher example (thanks, @nishan-soni!)
  • Corrected broken links on the automation landing page (thanks, @briandailey!)

Dagster Plus

  • [ui] Alert policy event tags no longer appear red and yellow outside of the policy notification history.

dg & Components (Preview)

  • Added suggestions to component model error messages when using built-in models for common classes such as AssetKey and AssetSpec.
  • dg list env now displays whether env vars are configured in each Dagster Plus scope.
  • Introduced Resolver.passthrough() to avoid processing fields on a component model.
  • ResolvedAssetKey is now exported from dagster.components.
  • dg init has been removed. dg scaffold project and dg scaffold workspace should be used instead.
  • Fixed an issue where dg dev failed with a temporarily file permissions error when running on Windows. Thanks @polivbr!