EphemeralWriteOnly: add secretAccessKeyWo

[BBBmau]

Release Note Template for Downstream PRs (will be copied)

See Write release notes for guidance.

bigquerydatatransfer: added `secret_access_key_wo` write-only field to `google_bigquery_data_transfer_config` resource

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 24 insertions(+), 7 deletions(-))
google-beta provider: Diff ( 2 files changed, 24 insertions(+), 7 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_bigquery_data_transfer_config (12 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_bigquery_data_transfer_config" "primary" {
  sensitive_params {
    secret_access_key_wo = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 1
Passed tests: 0
Skipped tests: 0
Affected tests: 1

Click here to see the affected service packages

• bigquerydatatransfer

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccBigqueryDataTransferConfig

Get to know how VCR tests work

[modular-magician]

🟢 Tests passed during RECORDING mode:
TestAccBigqueryDataTransferConfig [Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

---

🟢 All tests passed!

View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 24 insertions(+), 7 deletions(-))
google-beta provider: Diff ( 2 files changed, 24 insertions(+), 7 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_bigquery_data_transfer_config (12 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_bigquery_data_transfer_config" "primary" {
  sensitive_params {
    secret_access_key    = # value needed
    secret_access_key_wo = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 1
Passed tests: 1
Skipped tests: 0
Affected tests: 0

Click here to see the affected service packages

• bigquerydatatransfer

🟢 All tests passed!

View the build log

[wj-chen]

Thanks for the PR, could you share more about this change? Is it in response to an issue or bug or feature request?

[BBBmau]

@wj-chen this comes from supporting ephemeral write-only attributes, however I am running into a blocker that you could potentially help with. The context:
it looks like the field sensitive_params.secret_access_key doesn't work anymore due to params keys being updated (assuming that the secret_access_key was in the params list in the first place).
Currently looking for some sort of record of this, whether or not it was set in the past. any help would be appreciated!
This comes from doing some testing with the field itself as it's a write-only attribute candidate. Wasn't able to use it normally which led to the discovery.

https://cloud.google.com/bigquery/docs/cloud-storage-transfer#bq

Edit: this has been resolved after coming across: https://cloud.google.com/bigquery/docs/s3-transfer#bq

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 33 insertions(+), 7 deletions(-))
google-beta provider: Diff ( 2 files changed, 33 insertions(+), 7 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 8 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_bigquery_data_transfer_config (12 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_bigquery_data_transfer_config" "primary" {
  sensitive_params {
    secret_access_key    = # value needed
    secret_access_key_wo = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 0
Passed tests: 0
Skipped tests: 0
Affected tests: 0

Click here to see the affected service packages

• bigquerydatatransfer

🔴 Errors occurred during REPLAYING mode. Please fix them to complete your PR.

View the build log

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 35 insertions(+), 7 deletions(-))
google-beta provider: Diff ( 2 files changed, 35 insertions(+), 7 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 8 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_bigquery_data_transfer_config (12 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_bigquery_data_transfer_config" "primary" {
  sensitive_params {
    secret_access_key    = # value needed
    secret_access_key_wo = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 0
Passed tests: 0
Skipped tests: 0
Affected tests: 0

Click here to see the affected service packages

• bigquerydatatransfer

🔴 Errors occurred during REPLAYING mode. Please fix them to complete your PR.

View the build log

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 42 insertions(+), 7 deletions(-))
google-beta provider: Diff ( 2 files changed, 42 insertions(+), 7 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 15 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_bigquery_data_transfer_config (12 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_bigquery_data_transfer_config" "primary" {
  sensitive_params {
    secret_access_key    = # value needed
    secret_access_key_wo = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 0
Passed tests: 0
Skipped tests: 0
Affected tests: 0

Click here to see the affected service packages

• bigquerydatatransfer

🔴 Errors occurred during REPLAYING mode. Please fix them to complete your PR.

View the build log

[wj-chen]

@wj-chen this comes from supporting ephemeral write-only attributes, however I am running into a blocker that you could potentially help with. The context: it looks like the field sensitive_params.secret_access_key doesn't work anymore due to params keys being updated (assuming that the secret_access_key was in the params list in the first place). Currently looking for some sort of record of this, whether or not it was set in the past. any help would be appreciated! This comes from doing some testing with the field itself as it's a write-only attribute candidate. Wasn't able to use it normally which led to the discovery. https://cloud.google.com/bigquery/docs/cloud-storage-transfer#bq

Edit: this has been resolved after coming across: https://cloud.google.com/bigquery/docs/s3-transfer#bq

Glad you were able to find the documentation on the field. Is the support for write-only attributes being rolled out to all google-provider (and other providers)? Do I understand correctly that the recommendation is to use the write-only attribute and deprecate the other one over time? I took only a quick glance at https://developer.hashicorp.com/terraform/plugin/sdkv2/resources/write-only-arguments so maybe it's covered there but what happens if user wants to change the value if it's not part of the planning phase?

[BBBmau]

@wj-chen this comes from supporting ephemeral write-only attributes, however I am running into a blocker that you could potentially help with. The context: it looks like the field sensitive_params.secret_access_key doesn't work anymore due to params keys being updated (assuming that the secret_access_key was in the params list in the first place). Currently looking for some sort of record of this, whether or not it was set in the past. any help would be appreciated! This comes from doing some testing with the field itself as it's a write-only attribute candidate. Wasn't able to use it normally which led to the discovery. https://cloud.google.com/bigquery/docs/cloud-storage-transfer#bq
Edit: this has been resolved after coming across: https://cloud.google.com/bigquery/docs/s3-transfer#bq

Glad you were able to find the documentation on the field. Is the support for write-only attributes being rolled out to all google-provider (and other providers)? Do I understand correctly that the recommendation is to use the write-only attribute and deprecate the other one over time? I took only a quick glance at https://developer.hashicorp.com/terraform/plugin/sdkv2/resources/write-only-arguments so maybe it's covered there but what happens if user wants to change the value if it's not part of the planning phase?

Currently there is no way for Terraform to allow you to update a write-only value. The design perspective is that every ephemeral value is different on every action performed by terraform.

The current work around though that would allow this would be a sidecar field that's meant for triggering an update on an ephemeral value assuming you are in a case where the ephemeral write-only value itself is in a stable state that doesn't change unless manually triggered by a user. Hope this was clear.

On a side note I'll be moving this to ready for review as the secret_access_key doesn't contain tests due to the nature of the field. I've implemented it in a way that replicates the field when not being used in a amazon_s3.

the tfc_conversion can be ignored for now as it's being addressed in this PR on downstream tgc repo: GoogleCloudPlatform/terraform-google-conversion#3412

we required the use of sdk/v2 v2.26.0 while tgc uses v2.23.0. The linked PR would bump the needed dependency.

[wj-chen]

Is it a convention that these should always be suffixed "_wo"? Can we communicate it more in the YAML comment or somewhere so people know the name takes part in this logic?

[wj-chen]

Rest of the PR LGTM. Wanted to see if you have ideas to make the sp = sp[:len(sp)-3] part more future-proof.

[BBBmau]

yeah it's convention to mark write-only attributes with the _wo suffix. I can include this as a comment.

[wj-chen]

A comment would help, thanks.

[wj-chen]

@wj-chen this comes from supporting ephemeral write-only attributes, however I am running into a blocker that you could potentially help with. The context: it looks like the field sensitive_params.secret_access_key doesn't work anymore due to params keys being updated (assuming that the secret_access_key was in the params list in the first place). Currently looking for some sort of record of this, whether or not it was set in the past. any help would be appreciated! This comes from doing some testing with the field itself as it's a write-only attribute candidate. Wasn't able to use it normally which led to the discovery. https://cloud.google.com/bigquery/docs/cloud-storage-transfer#bq
Edit: this has been resolved after coming across: https://cloud.google.com/bigquery/docs/s3-transfer#bq

Glad you were able to find the documentation on the field. Is the support for write-only attributes being rolled out to all google-provider (and other providers)? Do I understand correctly that the recommendation is to use the write-only attribute and deprecate the other one over time? I took only a quick glance at https://developer.hashicorp.com/terraform/plugin/sdkv2/resources/write-only-arguments so maybe it's covered there but what happens if user wants to change the value if it's not part of the planning phase?

Currently there is no way for Terraform to allow you to update a write-only value. The design perspective is that every ephemeral value is different on every action performed by terraform.

The current work around though that would allow this would be a sidecar field that's meant for triggering an update on an ephemeral value assuming you are in a case where the ephemeral write-only value itself is in a stable state that doesn't change unless manually triggered by a user. Hope this was clear.

On a side note I'll be moving this to ready for review as the secret_access_key doesn't contain tests due to the nature of the field. I've implemented it in a way that replicates the field when not being used in a amazon_s3.

the tfc_conversion can be ignored for now as it's being addressed in this PR on downstream tgc repo: GoogleCloudPlatform/terraform-google-conversion#3412

we required the use of sdk/v2 v2.26.0 while tgc uses v2.23.0. The linked PR would bump the needed dependency.

Thanks. I'm trying to understand from the user side, if they create a resource with a secret_access_key_wo, then regenerate an access key, and want to update their Terraform config with the new key, what will happen when the field is write-only? Sounds like it just doesn't update, unless it's marked as requires_replace?

[BBBmau]

Thanks. I'm trying to understand from the user side, if they create a resource with a secret_access_key_wo, then regenerate an access key, and want to update their Terraform config with the new key, what will happen when the field is write-only? Sounds like it just doesn't update, unless it's marked as requires_replace?

The idea is that write-only attributes are different during every action performed by Terraform. Although some values may stay stable it doesn't allow for identifying whether an update of a value occurred or not. Since write-only attributes can accept ephemeral values, write-only attribute configuration values are not expected to be consistent between plan and apply.

So for a typical user they wouldn't need to handle any require_replace as it's already expected through Terraform's eyes that the value for this is expected to not remain consistent. @wj-chen

[wj-chen]

Thanks. I'm trying to understand from the user side, if they create a resource with a secret_access_key_wo, then regenerate an access key, and want to update their Terraform config with the new key, what will happen when the field is write-only? Sounds like it just doesn't update, unless it's marked as requires_replace?

The idea is that write-only attributes are different during every action performed by Terraform. Although some values may stay stable it doesn't allow for identifying whether an update of a value occurred or not. Since write-only attributes can accept ephemeral values, write-only attribute configuration values are not expected to be consistent between plan and apply.

So for a typical user they wouldn't need to handle any require_replace as it's already expected through Terraform's eyes that the value for this is expected to not remain consistent. @wj-chen

I see thanks. This PR looks good to me, but two more questions on the best use case of write-only fields in general:

• Since an AWS secret access key stays unchanged unless the user rotates it, is the use of write-only in this case more about skipping storing the value and removing diffs, and less about the value being ephemeral?
• Does changing a field to/from write-only considered a breaking change?

[BBBmau]

I see thanks. This PR looks good to me, but two more questions on the best use case of write-only fields in general:

• Since an AWS secret access key stays unchanged unless the user rotates it, is the use of write-only in this case more
  about skipping storing the value and removing diffs, and less about the value being ephemeral?

The use-case is more to prevent sensitive information from being stored in state. You can prevent values from beiung shown in the plan with Sensitive: true but that doesn't prevent the sensitive info from being shown in terraform.tf.state. This is where WriteOnly comes in which allows you to use sensitive information when it's necessary while also not needing to store it.

• Does changing a field to/from write-only considered a breaking change?

It does, however we intend to keep both the WriteOnly and non-Writeonly included with their being Validation added to push users to use the new WriteOnly Attribute. The non-writeOnly attribute would be removed in an upcoming major release. This is an example of the validation being shown to users. PR would support Resource Validation to support this: #13048 [this must be merged before we merge this PR]

[wj-chen]

I see thanks. This PR looks good to me, but two more questions on the best use case of write-only fields in general:

• Since an AWS secret access key stays unchanged unless the user rotates it, is the use of write-only in this case more
  about skipping storing the value and removing diffs, and less about the value being ephemeral?

The use-case is more to prevent sensitive information from being stored in state. You can prevent values from beiung shown in the plan with Sensitive: true but that doesn't prevent the sensitive info from being shown in terraform.tf.state. This is where WriteOnly comes in which allows you to use sensitive information when it's necessary while also not needing to store it.

• Does changing a field to/from write-only considered a breaking change?

It does, however we intend to keep both the WriteOnly and non-Writeonly included with their being Validation added to push users to use the new WriteOnly Attribute. The non-writeOnly attribute would be removed in an upcoming major release. This is an example of the validation being shown to users. PR would support Resource Validation to support this: #13048 [this must be merged before we merge this PR]

Thank you for the additional context. I left two in-line comments but the change looks good otherwise.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 82 insertions(+), 7 deletions(-))
google-beta provider: Diff ( 2 files changed, 82 insertions(+), 7 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 25 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_bigquery_data_transfer_config (12 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_bigquery_data_transfer_config" "primary" {
  sensitive_params {
    secret_access_key    = # value needed
    secret_access_key_wo = # value needed
  }
  sensitive_params_wo_version = # value needed
}

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 88 insertions(+), 7 deletions(-))
google-beta provider: Diff ( 3 files changed, 88 insertions(+), 7 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 11 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_bigquery_data_transfer_config (12 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_bigquery_data_transfer_config" "primary" {
  sensitive_params {
    secret_access_key    = # value needed
    secret_access_key_wo = # value needed
  }
  sensitive_params_wo_version = # value needed
}

[modular-magician]

Tests analytics

Total tests: 1
Passed tests: 0
Skipped tests: 0
Affected tests: 1

Click here to see the affected service packages

• bigquerydatatransfer

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccBigqueryDataTransferConfig

Get to know how VCR tests work

[modular-magician]

🔴 Several tests terminated during RECORDING mode.

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 88 insertions(+), 7 deletions(-))
google-beta provider: Diff ( 3 files changed, 88 insertions(+), 7 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 11 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_bigquery_data_transfer_config (12 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_bigquery_data_transfer_config" "primary" {
  sensitive_params {
    secret_access_key    = # value needed
    secret_access_key_wo = # value needed
  }
  sensitive_params_wo_version = # value needed
}

[modular-magician]

Tests analytics

Total tests: 1
Passed tests: 0
Skipped tests: 0
Affected tests: 1

Click here to see the affected service packages

• bigquerydatatransfer

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccBigqueryDataTransferConfig

Get to know how VCR tests work

[modular-magician]

🟢 Tests passed during RECORDING mode:
TestAccBigqueryDataTransferConfig [Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

---

🟢 All tests passed!

View the build log or the debug log for each test

[BBBmau]

recent pushes were just rebasing