mirrord does not correctly pass through TLS certificate files for Granian

[benglewis]

Bug Description

When using Granian, the TLS certificate arguments are incorrectly handled and the files are not loaded from the remote pod in the Kubernetes cluster

Steps to Reproduce

1. Create a new Python project with a Dockerfile using Granian.
2. Add TLS certificates to the Granian server and inject those certificates using Kubernetes secrets (e.g. via cert-manager) into the deployment.
3. Create a local debug configuration to use with mirrord which has the TLS certificate arguments with the same path as they are injected into the Kubernetes pod.
4. Configure mirrord using the .mirrord/mirrord.json file and add the two TLS certificate (cert and private key) files to the fs block in the read_only section.

Backtrace

File "$PROJECT/.pixi/envs/dev/lib/python3.13/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
                     "__main__", mod_spec)
  File "$PROJECT/.pixi/envs/dev/lib/python3.13/runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ~~~~^^^^^^^^^^^^^^^^^^^
  File "$HOME/.cursor/extensions/ms-python.debugpy-2025.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 71, in <module>
    cli.main()
    ~~~~~~~~^^
  File "$HOME/.cursor/extensions/ms-python.debugpy-2025.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 501, in main
    run()
    ~~~^^
  File "$HOME/.cursor/extensions/ms-python.debugpy-2025.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 384, in run_module
    run_module_as_main(options.target, alter_argv=True)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$HOME/.cursor/extensions/ms-python.debugpy-2025.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 228, in _run_module_as_main
    return _run_code(code, main_globals, None, "__main__", mod_spec)
  File "/Users/blewis/.cursor/extensions/ms-python.debugpy-2025.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
    exec(code, run_globals)
    ~~~~^^^^^^^^^^^^^^^^^^^
  File "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/granian/__main__.py", line 4, in <module>
    entrypoint()
    ~~~~~~~~~~^^
  File "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/granian/cli.py", line 474, in entrypoint
    cli(auto_envvar_prefix='GRANIAN')
    ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
  File "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/granian/cli.py", line 400, in cli
    server = Server(
        app,
    ...<62 lines>...
        pid_file=pid_file,
    )
  File "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/granian/server/common.py", line 189, in __init__
    self.build_ssl_context(ssl_cert, ssl_key, ssl_key_password, ssl_ca, ssl_crl or [], ssl_client_verify)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/granian/server/common.py", line 216, in build_ssl_context
    ctx.load_cert_chain(str(cert.resolve()), str(key.resolve()), password)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory

mirrord layer logs

2025-07-06T11:34:26.647690Z TRACE ThreadId(01) mirrord_layer::exec_utils: dlopen called on path Success("$PROJECT/.pixi/envs/dev/lib/libcrypto.3.dylib").
2025-07-06T11:34:26.651299Z TRACE ThreadId(01) mirrord_layer::file::hooks: path Success(
    "$PROJECT/.pixi/envs/dev/ssl/openssl.cnf",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
2025-07-06T11:34:26.886425Z  INFO ThreadId(01) mirrord_layer::error: libc error (doesn't indicate a problem) >> ResponseError(
    RemoteIO(
        RemoteIOError {
            raw_os_error: Some(
                2,
            ),
            kind: NotFound,
        },
    ),
)
2025-07-06T11:34:27.116607Z  INFO ThreadId(01) mirrord_layer::error: libc error (doesn't indicate a problem) >> ResponseError(
    RemoteIO(
        RemoteIOError {
            raw_os_error: Some(
                2,
            ),
            kind: NotFound,
        },
    ),
)
2025-07-06T11:34:27.116955Z TRACE ThreadId(01) mirrord_layer::file::hooks: path Success(
    "/System/Volumes/Data/$DOCKER_PROJECT_PATH/certs/tls.crt",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
2025-07-06T11:34:27.280617Z  INFO ThreadId(01) mirrord_layer::error: libc error (doesn't indicate a problem) >> ResponseError(
    RemoteIO(
        RemoteIOError {
            raw_os_error: Some(
                2,
            ),
            kind: NotFound,
        },
    ),
)
2025-07-06T11:34:27.283222Z TRACE ThreadId(01) mirrord_layer::file::hooks: path Success(
    "$PROJECT/.pixi/envs/dev/lib/python3.13/runpy.py",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
2025-07-06T11:34:27.284322Z TRACE ThreadId(01) mirrord_layer::file::hooks: path Success(
    "$HOME/.cursor/extensions/ms-python.debugpy-2025.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
2025-07-06T11:34:27.285142Z TRACE ThreadId(01) mirrord_layer::file::hooks: path Success(
    "$HOME/.cursor/extensions/ms-python.debugpy-2025.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
2025-07-06T11:34:27.286191Z TRACE ThreadId(01) mirrord_layer::file::hooks: path Success(
    "$HOME/.cursor/extensions/ms-python.debugpy-2025.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
2025-07-06T11:34:27.287063Z TRACE ThreadId(01) mirrord_layer::file::hooks: path Success(
    "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/granian/__main__.py",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
2025-07-06T11:34:27.287717Z TRACE ThreadId(01) mirrord_layer::file::hooks: path Success(
    "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/granian/cli.py",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
2025-07-06T11:34:27.289959Z TRACE ThreadId(01) mirrord_layer::file::hooks: path Success(
    "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/click/core.py",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}
2025-07-06T11:34:27.293453Z TRACE ThreadId(01) mirrord_layer::file::hooks: path Success(
    "$PROJECT/.pixi/envs/dev/lib/python3.13/site-packages/granian/server/common.py",
) | open_options OpenOptionsInternal {
    read: true,
    write: false,
    append: false,
    truncate: false,
    create: false,
    create_new: false,
}

mirrord intproxy logs

mirrord agent logs

mirrord config

{
  "accept_invalid_certificates": false,
  "feature": {
    "network": {
      "incoming": "steal",
      "outgoing": {
        "filter": {
          "local": "tcp://127.0.0.1:5173"
        }
      }
    },
    "fs": {
      "mode": "read",
      "read_only": [
        "/etc/secrets/DB_SECRET_KEY",
        "{OUR_PROJECT_FOLDER_NAME}/certs/.+",
        "{OUR_PROJECT_FOLDER_NAME}/client/build/config/config.json",
        "{HOME_FOLDER_NAME}/.mlflow/credentials"
      ],
      "mapping": {
        "^(?:/Users|/home)/(?<user>\\S+)/.mlflow/credentials": "{HOME_FOLDER_NAME}/.mlflow/credentials"
      }
    },
    "env": {
      "override": {"RUNENV": "mirrord"}
    }
  },
  "agent": {
    "namespace": "default",
    "communication_timeout": 3600
  },
  "telemetry": false,
  "target": "deployment/{DEPLOYMENT_NAME}"
}

mirrord CLI version

3.124.2

mirrord-agent version

No response

mirrord-operator version (if relevant)

No response

plugin kind and version (if relevant)

3.66.0

Your operating system and version

macOS 15.5

Local process

python -m granian

Local process version

Python 3.13 Granian 2.4.0

Additional Info

No response

[linear]

MBE-1074 mirrord does not correctly pass through TLS certificate files for Granian

[benglewis]

@eyalb181 If there is any more information that I can provide that would be helpful, please let me know 🙏