dataset features (merging, loading, streaming from remote)

[hawkoli1987]

As a user of both llm-foundry (based on mosaic composer) and nemo (with megatron core), the following are some merits on mosaic dataset which would be great if also added into nemo. Like to invite some thoughts on this.

1. the mosaic dataset index is a lightweight (in ~ MB size), the mcore .idx file is typically in GB sizes. The key reason I guess is that mosaic dataset truncate/pad the sequences to target seq_len during preprocessing stage, hence it is not required to save each individual document length into the idx.
2. the mosaic dataset index is in json, vs mcore .idx is in binary. The benefit is that:

• index file is human readable
• operations like dataset merging or splitting can be much easily done
• it saves on CPU/RAM when loading a large number of datasets simulatenous in a training.

3. the mosaic dataset content file is in equal sized partitions of .mds files, vs mcore .bin is monolithic. The problem with a single large size .bin file is that it makes inter-cluster data transfer more risky, and potentially hitting the single file size limit (e.g. AWS S3 limits of 5TB)
4. this an option to remote streaming the dataset druing training, which can be helpful when the local storage is limited.