Skip to content

Commit

Permalink
ref(*)!: Reorganizes the blobstore interface
Browse files Browse the repository at this point in the history
This is a relatively simple PR that mostly restructures things. This
removes the container interface because there wasn't much you could do
with it by itself, it always had to be used in conjunction with the
blobstore interface. I also got rid of all of the type aliases for
strings to make the types interface cleaner, as well as follow what we've
been doing in the other wasi cloud interfaces with avoiding type aliases.

Additionally, I updated the timestamps to use the wasi-clock timestamps
instead so we were using a concrete type. That was the only major change
here, but it does solve an outstanding issue we have.

Closes WebAssembly#7

Signed-off-by: Taylor Thomas <[email protected]>
  • Loading branch information
thomastaylor312 authored and rvolosatovs committed Aug 16, 2024
1 parent 18f1288 commit 13c9422
Show file tree
Hide file tree
Showing 15 changed files with 368 additions and 508 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ jobs:
- uses: actions/checkout@v2
- uses: WebAssembly/wit-abi-up-to-date@v17
with:
wit-bindgen: '0.18.0'
wit-bindgen: '0.22.0'
73 changes: 49 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Phase 1
- Jiaxiao Zhou
- Kevin Hoffman
- David Justice
- Dan Chiarlon
- Dan Chiarlone
- Taylor Thomas

### Phase 4 Advancement Criteria
Expand All @@ -23,28 +23,45 @@ Phase 1

## Table of Contents

- [Introduction](#introduction)
- [Goals](#goals)
- [Non-goals](#non-goals)
- [API walk-through](#api-walk-through)
- [Process Blob Contents](#process-blob-contents)
- [Write to a Blob Stream](#write-to-a-blob-stream)
- [List Objects within a Container](#list-objects-within-a-container)
- [Detailed Design Discussion](#detailed-design-discussion)
- [Handling Large Files](#handling-large-files)
- [Stakeholder Interest & Feedback](#stakeholder-interest--feedback)
- [WASI Blob Store](#wasi-blob-store)
- [Current Phase](#current-phase)
- [Champions](#champions)
- [Phase 4 Advancement Criteria](#phase-4-advancement-criteria)
- [Table of Contents](#table-of-contents)
- [Introduction](#introduction)
- [Goals](#goals)
- [Non-goals](#non-goals)
- [API walk-through](#api-walk-through)
- [Process Blob Contents](#process-blob-contents)
- [Write to a Blob Stream](#write-to-a-blob-stream)
- [List Objects within a Container](#list-objects-within-a-container)
- [Detailed Design Discussion](#detailed-design-discussion)
- [Handling Large Files](#handling-large-files)
- [Stakeholder Interest \& Feedback](#stakeholder-interest--feedback)

### Introduction

**Blob storage** is a type of data storage used for unstructured data such as images, videos, documents, backups, etc. Blob storage is also commonly referred to as _object storage_. The term **blob** is actually an acronym for **B**inary **L**arge **OB**ject but can be used to refer to all types of unstructured data.
**Blob storage** is a type of data storage used for unstructured data such as images, videos,
documents, backups, etc. Blob storage is also commonly referred to as _object storage_. The term
**blob** is actually an acronym for **B**inary **L**arge **OB**ject but can be used to refer to all
types of unstructured data.

Within the context of this proposal, blob storage refers to granting WebAssembly components access to a common abstraction of a blob store. Examples of blob storage services include [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/), [AWS S3](https://aws.amazon.com/s3/), or [Google Cloud Storage](https://cloud.google.com/storage), but can be anything that can be represented as unstructured binary data that conforms to the interface, including file systems.
Within the context of this proposal, blob storage refers to granting WebAssembly components access
to a common abstraction of a blob store. Examples of blob storage services include [Azure Blob
Storage](https://azure.microsoft.com/en-us/services/storage/blobs/), [AWS
S3](https://aws.amazon.com/s3/), or [Google Cloud Storage](https://cloud.google.com/storage), but
can be anything that can be represented as unstructured binary data that conforms to the interface,
including file systems.

### Goals

The primary goal of this API is to provide a common abstraction for blob storage services, so that WebAssembly components can be written to work with any implementation, without needing to know the details of the underlying service.
The primary goal of this API is to provide a common abstraction for blob storage services, so that
WebAssembly components can be written to work with any implementation, without needing to know the
details of the underlying service.

Additionally, components using this API will be unable to tell the difference between a blob storage service and a file system, allowing them to be written to work with either and will not need to configure the store within the component code.
Additionally, components using this API will be unable to tell the difference between a blob storage
service and a file system, allowing them to be written to work with either and will not need to
configure the store within the component code.

### Non-goals
The following is a list of goals explicitly out of scope for this API specification:
Expand All @@ -57,15 +74,17 @@ The following is a list of goals explicitly out of scope for this API specificat

### API walk-through

The following sections provide an overview of how this API might be used. Note that while the samples are in Rust, any language targetable by wasm components via code generation should work.
The following sections provide an overview of how this API might be used. Note that while the
samples are in Rust, any language targetable by wasm components via code generation should work.

#### Process Blob Contents
This example shows obtaining a reference to the container and the desired object within that container, and then using `read_into` in a loop to access the blob contents.
This example shows obtaining a reference to the container and the desired object within that
container, and then using `read_into` in a loop to access the blob contents.

```rust
// Count the number of lines in an object
// For simplicity, assume the object contains ascii text and lines end in '\n'
fn count_lines(store: &impl BlobStore, id: &ObjectId) -> Result<usize, Error> {
fn count_lines(store: &impl Blobstore, id: &ObjectId) -> Result<usize, Error> {
let mut stream = store.get_container(&id.container_name)?.read_object(&id.object_name)?;
let mut buf = [0u8; 4096];
let mut num_lines = 0;
Expand All @@ -77,12 +96,13 @@ fn count_lines(store: &impl BlobStore, id: &ObjectId) -> Result<usize, Error> {
```

#### Write to a Blob Stream
The following code sample shows how to obtain a reference to a container and a writable reference to a stream that will be stored in a blob.
The following code sample shows how to obtain a reference to a container and a writable reference to
a stream that will be stored in a blob.

```rust
// Download a file from an http url and save it to the blob store.
// When completed, returns metadata for the new object
fn download(url: &str, store: &impl BlobStore, id: &ObjectId) -> Result<ObjectMetadata, Error> {
fn download(url: &str, store: &impl Blobstore, id: &ObjectId) -> Result<ObjectMetadata, Error> {
let container = store.get_container(&id.container_name)?;
// retrieve a url via wasi-http fetch() method
// the http service hasn't been defined yet, but assume its fetch() method returns a readable stream.
Expand All @@ -106,7 +126,7 @@ The following code shows how to enumerate the objects within a container.
// suppose the "logs" container has objects with names that start with a timestamp, like "2022-01-01-12-00-00.log"
// for every day that activity occurred. To count the number of logs from january 2022, call:
// `count_objects_with_prefix(store, "logs", "2022-01")`
fn count_objects_with_prefix(store: &impl BlobStore, container_name: &str, prefix: &str) -> Result<usize,Error> {
fn count_objects_with_prefix(store: &impl Blobstore, container_name: &str, prefix: &str) -> Result<usize,Error> {
let container = store.get_container(container_name)?;
let names = container.list_objects()?;
let count = names.filter(|n| n.starts_with(prefix)).count();
Expand All @@ -120,11 +140,16 @@ See the [wit files](./wit)

#### Handling Large Files

Handling large files may require changes to the API that are not accounted for in this current proposal. If a component attempts to allocate more memory than the host is willing to give it, then the component could be terminated by the host runtime and the processing will fail.
Handling large files may require changes to the API that are not accounted for in this current
proposal. If a component attempts to allocate more memory than the host is willing to give it, then
the component could be terminated by the host runtime and the processing will fail.

Additionally, if a component spends too long processing a file, either processing one large blob or by processing many small blobs in a tight loop, then the component could again be shut off because it consumed too many resources or too much time.
Additionally, if a component spends too long processing a file, either processing one large blob or
by processing many small blobs in a tight loop, then the component could again be shut off because
it consumed too many resources or too much time.

How to handle this and whether the `callback` approach belongs in the blob store API or in a lower level [wasm-io API](https://github.com/WebAssembly/wasi-io/issues/31) is still under discussion.
How to handle this and whether the `callback` approach belongs in the blob store API or in a lower
level [wasm-io API](https://github.com/WebAssembly/wasi-io/issues/31) is still under discussion.

### Stakeholder Interest & Feedback

Expand Down
Empty file removed blobstore.abi.md
Empty file.
178 changes: 0 additions & 178 deletions blobstore.wit.md

This file was deleted.

Loading

0 comments on commit 13c9422

Please sign in to comment.