Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider length hint for outgoing values #34

Open
itowlson opened this issue Nov 25, 2024 · 0 comments
Open

Consider length hint for outgoing values #34

itowlson opened this issue Nov 25, 2024 · 0 comments

Comments

@itowlson
Copy link

Large blobs in Azure are uploaded as a series of blocks. A blob may contain a maximum of 50 000 blocks.

Therefore, the wasi-blobstore implementation cannot upload data as the guest writes it, because that could result in running out of blocks. That's fine, the implementation can just buffer the data until it has a good amount to write...

...but how can it determine a "good" amount? A safe buffer size for the largest of large blobs is 4GB, but that's a ridiculous amount of buffering for what is realistically an edge case! Or I might guess 1MB, which would let me upload blobs up to 50GB, but then someone is going to get mad at me when their 51GB upload fails.

A possible way to address this is to allow a length hint on outgoing values, either at construction time or when calling outgoing-value-write-body. This would allow the implementation to choose a buffer size large enough to keep the number of blocks within an allowable margin.

Additional thoughts:

  • I could cook up some sort of adaptive scheme. Say, start with a 1MB buffer, then double it every time until I hit the maximum buffer size. Nobody is going to complain about losing 48GB of their maximum 190TB. But as an implementer, this does not spark joy...
  • In many situations, letting an arbitrary guest fill a 4GB buffer is a one-way trip to a denial of service. So it's possible I'd want to limit the buffer size (and hence max blob size) anyway. On the other hand, in that case a length hint would still be useful so I could send an informative message when they asked for an excessive length.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant