Instaling mirrord-operator causes kubernetes terraform provider to start failing on CRD parsing

[owen-c]

Bug Description

After installing the mirrord-operator, all of our terraformed kubernetes_manifest resources start failing to plan. This is because the planning step for this resource requires reading all CRDs of which mirrord has installed a new one.

Something about the mirrord CRD is failing to be parsed by the official hashicorp terraform provider. We have plenty of other CRDs and none cause this behavior.

The error is pasted below, this occurs for every single kubernetes_manifest resource in our terraform.

I'm hoping something easy can be done with the CRD to avoid this issue, because potentially getting this fixed in the terraform provider seems more onerous.

I'm testing with the latest verison of the kubernetes_provider 2.37.1
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs

Error: Plugin error

   with kubernetes_manifest.servicemonitor_prometheus_prometheus,
   on karpenter-service-monitor.tf line 1, in resource "kubernetes_manifest" "servicemonitor_prometheus_prometheus":
    1: resource "kubernetes_manifest" "servicemonitor_prometheus_prometheus" {

 The plugin returned an unexpected error from plugin.(*GRPCProvider).UpgradeResourceState: rpc error: code = Unknown desc = failed to determine resource type ID: cannot get OpenAPI foundry: failed construct OpenAPI foundry: failed to parse spec: failed to unmarshal property "paths" (*map[string]*openapi2.PathItem): failed to unmarshal property "get" (*openapi2.Operation): failed to unmarshal property "responses" (*map[string]*openapi2.Response): failed to unmarshal property "schema" (*openapi3.SchemaRef): failed to unmarshal property "properties" (*openapi3.Schemas): failed to unmarshal property "items" (*openapi3.SchemaRef): failed to unmarshal property "properties" (*openapi3.Schemas): failed to unmarshal property "properties" (*openapi3.Schemas): failed to unmarshal property "items" (*openapi3.SchemaRef): failed to unmarshal property "items" (*openapi3.SchemaRef): failed to unmarshal extension properties: json: cannot unmarshal array into Go value of type map[string]json.RawMessage ([{"type":"string"},{"description":"Auto-generated derived type for CopyTargetSpec via `CustomResource`","type":"object","required":["spec"],"properties":{"spec":{"description":"This resource represents a copy pod created from an existing [`Target`] (operator's copy pod feature).","type":"object","required":["scale_down","target"],"properties":{"idle_ttl":{"description":"How long should the operator keep this pod alive after its creation. The pod is deleted when this timout has expired and there are no connected clients.","type":"integer","format":"uint32","minimum":0},"scale_down":{"description":"Should the operator scale down target deployment to 0 while this pod is alive. Ignored if [`Target`] is not [`Target::Deployment`].","type":"boolean"},"split_queues":{"description":"Split queues client side configuration.","type":"object","additionalProperties":{"description":"Amazon Simple Queue Service.","type":"object","properties":{"message_filter":{"description":"A filter is a mapping between message header names and regexes they should match. The local application will only receive messages that match **all** of the given patterns. This means, only messages that have **all** of the headers in the filter, with values of those headers matching the respective patterns.","type":"object","additionalProperties":{"type":"string"}},"queue_type":{"type":"string","enum":["Kafka"]}}}},"target":{"description":"<!--${internal}--> [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/).","type":"object","properties":{"container":{"type":"string"},"cron_job":{"type":"string"},"deployment":{"description":"<!--${internal}--> Deployment to mirror.","type":"string"},"job":{"type":"string"},"pod":{"description":"<!--${internal}--> Pod to mirror.","type":"string"},"replica_set":{"type":"string"},"rollout":{"description":"<!--${internal}--> Rollout to mirror.","type":"string"},"service":{"type":"string"},"stateful_set":{"type":"string"}},"additionalProperties":false}}},"status":{"description":"This is the `status` field for [`CopyTargetCrd`].","type":"object","required":["creator_session"],"properties":{"creator_session":{"description":"The session object of the original session that created this CopyTarget","type":"object","required":["duration_secs","target","user"],"properties":{"duration_secs":{"type":"integer","format":"uint64","minimum":0},"id":{"type":"string"},"locked_ports":{"type":"array","items":{"type":"array","maxItems":3,"minItems":3,"items":[{"type":"integer","format":"uint16","minimum":0},{"type":"string"},{"type":"string"}]}},"namespace":{"type":"string"},"sqs":{"type":"array","items":{"description":"Auto-generated derived type for MirrordSqsSessionSpec via `CustomResource`","type":"object","required":["spec"],"properties":{"spec":{"description":"The operator creates this object when a user runs mirrord against a target that is a queue consumer.","type":"object","required":["queueConsumer","queueFilters","sessionId"],"properties":{"queueConsumer":{"description":"The target of this session.","type":"object","required":["name","workloadType"],"properties":{"container":{"description":"If a container is not specified, the workload queue registry will apply to every run that targets any of the workload's containers.","type":"string"},"name":{"type":"string"},"workloadType":{"description":"A workload that is a consumer of a queue that is being split.","type":"string","enum":["Deployment","Rollout"]}}},"queueFilters":{"description":"For each queue_id, a mapping from attribute name, to attribute value regex. The queue_id for a queue is determined at the queue registry. It is not (necessarily) The name of the queue on AWS.","type":"object","additionalProperties":{"type":"object","additionalProperties":{"type":"string"}}},"sessionId":{"description":"The id of the mirrord exec session, from the operator.","type":"string"}}},"status":{"description":"SQS operator sets this status before it starts registering filters, so that if anything fails during the registration of filters, we have all the queues we need to delete on cleanup.","type":"object","properties":{"CleanupError":{"type":"object","required":["error"],"properties":{"details":{"type":"object","required":["envUpdates","queueNames"],"properties":{"envUpdates":{"description":"Env var name -> old and new queue names.","type":"object","additionalProperties":{"type":"object","required":["originalName","outputName"],"properties":{"originalName":{"type":"string"},"outputName":{"type":"string"}}}},"queueNames":{"description":"Queue ID -> old and new queue names.","type":"object","additionalProperties":{"type":"object","required":["originalName","outputName"],"properties":{"originalName":{"type":"string"},"outputName":{"type":"string"}}}}}},"error":{"description":"Representation of Sqs errors for the status of SQS session resources.","type":"object","required":["reason","statusCode"],"properties":{"reason":{"description":"Human-readable explanation of what went wrong.","type":"string"},"statusCode":{"description":"HTTP code for operator response.","type":"integer","format":"uint16","minimum":0}}}}},"Ready":{"type":"object","required":["envUpdates","queueNames"],"properties":{"envUpdates":{"description":"Env var name -> old and new queue names.","type":"object","additionalProperties":{"type":"object","required":["originalName","outputName"],"properties":{"originalName":{"type":"string"},"outputName":{"type":"string"}}}},"queueNames":{"description":"Queue ID -> old and new queue names.","type":"object","additionalProperties":{"type":"object","required":["originalName","outputName"],"properties":{"originalName":{"type":"string"},"outputName":{"type":"string"}}}}}},"RegisteringFilters":{"type":"object","required":["envUpdates","queueNames"],"properties":{"envUpdates":{"description":"Env var name -> old and new queue names.","type":"object","additionalProperties":{"type":"object","required":["originalName","outputName"],"properties":{"originalName":{"type":"string"},"outputName":{"type":"string"}}}},"queueNames":{"description":"Queue ID -> old and new queue names.","type":"object","additionalProperties":{"type":"object","required":["originalName","outputName"],"properties":{"originalName":{"type":"string"},"outputName":{"type":"string"}}}}}},"StartError":{"description":"Representation of Sqs errors for the status of SQS session resources.","type":"object","required":["reason","statusCode"],"properties":{"reason":{"description":"Human-readable explanation of what went wrong.","type":"string"},"statusCode":{"description":"HTTP code for operator response.","type":"integer","format":"uint16","minimum":0}}},"Starting":{"type":"object","required":["start_time_utc"],"properties":{"start_time_utc":{"type":"string"}}}},"additionalProperties":false}}}},"target":{"type":"string"},"user":{"type":"string"},"user_id":{"type":"string"}}}}}}}])

Steps to Reproduce

1. Use terraform to manage any kubernetes_manifest resource
2. Install mirrord-operator
3. Try to plan the kubernetes_manifest resource
4. Fail

Backtrace

mirrord layer logs

mirrord intproxy logs

mirrord agent logs

mirrord config

mirrord CLI version

No response

mirrord-agent version

No response

mirrord-operator version (if relevant)

1.29.0

plugin kind and version (if relevant)

No response

Your operating system and version

Mac 15.5

Local process

N/A

Local process version

No response

Additional Info

No response

[linear]

MBE-1054 Instaling mirrord-operator causes kubernetes terraform provider to start failing on CRD parsing

[aviramha]

hey,
It's a known issue with Terraform, we're trying to figure out how to solve it (the schema seems to be valid if we run it using other validation tools)
For now, you can solve it by having this in your values.yaml:

operator:
  extraEnv:
    OPERATOR_OPENAPI_V2_ENABLE: 'false'

Apologies for the inconvenience.

[owen-c]

Thank you! That worked for me.