[C++] Add option to not create parent directory with S3 `delete_file`

[dacort]

Describe the enhancement requested

Today, when using delete_file with S3FileSystem, it will create an empty file if there's no "directory".

Example:

Before:

aws s3 ls s3://bucket/prefix/

2023-06-23 11:43:53        668 prefix/01/file1.json.gz
2023-06-23 11:35:24       1438 prefix/01/file2.json.gz
2023-06-23 10:47:18        819 prefix/01/file3.json.gz

Run S3FileSystem::delete_file on file1.json.gz

After:

2023-06-23 11:51:02          0 prefix/01/
2023-06-23 11:35:24       1438 prefix/01/file2.json.gz
2023-06-23 10:47:18        819 prefix/01/file3.json.gz

This is due to this line that calls EnsureParentExists.

Unfortunately, this results in 0-byte files getting created on S3, even if there are other objects in the "directory". It seems like this was somewhat intentional based on this comment in the original PR.

That said, I'd love for the ability do disable this. delete_file shouldn't create additional files/directories in S3.

Component(s)

Other

[rmoff]

+1

I have just hit the same issue. Even with other objects present in the same directory, using DeleteFile from the R client writes an empty directory marker file.
IMHO this behaviour should either be a flag, or be smarter—check for other objects in the same path before taking it upon itself to create this file.

[westonpace]

This would be a change in behavior so I think we probably shouldn't change our default here. At the moment S3FileSystem is doing its best to have the S3 storage act like a normal filesystem and this is a part of that. So users may have come to rely on this behavior. However, we (as in arrow-c++) do not rely on this function at all (dataset writer does do some deletes but it only uses DeleteDirContents).

I do agree that there should be a way to avoid this. I think there are a few reasonable approaches that could be taken here if someone has time to work on this.

check for other objects in the same path before taking it upon itself to create this file.

Perhaps S3Options could have a setting to control whether marker directories are created or not (by any operation)

Of course, it would also be possible to create a very specific setting (e.g. S3Options::create_missing_dirs_on_delete)

I don't think adding any new flags to the delete method itself would be a good idea. There are others out there with their own custom implementations of FileSystem and so any changes to that interface incur additional work.

[dacort]

we probably shouldn't change our default here

Agree on this and makes sense to me given the scope of Arrow. S3Options does seem like a reasonable place for this.

If I get a chance, I may poke around with it, but for now I'm creating another S3 client and utilizing batch deletes via DeleteObjects as I'm almost always deleting a bunch of files anyway.

[pitrou]

I really think this is a bug in our implementation. There is no way that a method which claims to delete a file should attempt to create its parent directory if it didn't exist in the first place, so this should be fixed instead of adding an option.

[llama90]

take

[felipecrv]

I really think this is a bug in our implementation. There is no way that a method which claims to delete a file should attempt to create its parent directory if it didn't exist in the first place, so this should be fixed instead of adding an option.

If an object is deleted and an empty directory marker doesn't exist, the deletion of the file will imply the deletion of the directory.