Usage select by index in aws_dynamodb_partiql lead to ValidationError

[YuriiYa]

While trying to search by index

branch:
  request_map: |-
    root.sqlQuery = "SELECT neededId FROM \"table-name\".\"table-index\" WHERE id = '1234'"
  processors:
    - aws_dynamodb_partiql:
        query: '${! this.sqlQuery }'
        unsafe_dynamic_query: true
        args_mapping: |-
            root = []

Getting:
Error while getting data from table-name: failed to process statement (ValidationError): Select statements within BatchExecuteStatement must specify the primary key in the where clause.

Note:
The configuration of index:

• Global Secondary Index
• Partition key: id
• sort key: - (meaning NO sort key, table is pretty simple)
• Projected attributes: Include: neededId

So I am having only one partition key for index and as sort key is not mandatory(and I don't have it at all), I should be able to search by index using only index partition key.

And query in DynamoDB.PariQL editor works successfully

SELECT neededId  FROM "table-name"."table-index"
where id = '1234'

Used similar sql with parametrized args_mapping - having same error
Though while I am trying to access table - everything works correctly.

Didn't find any examples of searching by index either in redpanda connect repo tests or whenever in web.
Any suggestions or workarounds?
It seems like DynamoDB error "Select statements within BatchExecuteStatement must specify the primary key in the where clause. "

[mihaitodor]

Hey @YuriiYa, sorry for the late reply! Can you help me replicate this issue? I'm not super-familiar with DynamoDB. Here's what I have so far:

$ docker run --name localstack --rm -it -p 4566:4566 -e GATEWAY_LISTEN=0.0.0.0:4566 localstack/localstack
$ docker exec -it localstack awslocal dynamodb create-table --table-name benthos --key-schema AttributeName=id,KeyType=HASH --attribute-definitions AttributeName=id,AttributeType=S --billing-mode PAY_PER_REQUEST --region ap-south-1
$ docker exec -it localstack awslocal dynamodb put-item --table-name benthos --item '{"id":{"S":"foo"}}' --region ap-south-1

input:
  generate:
    count: 1
    mapping: root = ""

  processors:
    - aws_dynamodb_partiql:
        region: ap-south-1
        endpoint: http://localhost:4566
        credentials:
          id: xxxxx
          secret: xxxxx
          token: xxxxx
        query: "SELECT * FROM benthos where id = ?"
        args_mapping: |-
            root = [{"S": "foo"}]
    - log:
        message: ${! error() }

$ rpk connect run test.yaml
...
INFO null                                          @service=redpanda-connect custom_source=true label="" path=root.input.processors.1
{"id":{"S":"foo"}}

Can you please adjust this example to reproduce the error you're seeing?