ensure continuous recording

[Remi-Gau]

ensures that the file generated have continuous recording

[julia-pfarr]

What does that mean exactly? I'm sorry, I've always had troubles understanding this. Does that have to do with the timestamps? The samples do have continuous recording (=_recording-_physio.tsv.gz file). The events (=_recording-_physioevents.tsv.gz) does not. So for the events we would have to add extra "empty" rows that the timestamp is continous?

Is that what you mean? (asking because I'm currently working on the code for the _physioevents.tsv.gz file)

[Remi-Gau]

Yes the idea would be to pad physio.tsv.gz with rows of n/a for missing time stamps.

[julia-pfarr]

Ok, yes, I discussed this in our last meeting with Oscar and Martin. So the data (=physio.tsv.gz) should have continuous recording and I still need to modify the edf2bids code that it checks for continuity and if not fill with n/a rows. The physioevents file however (=physioevents.tsv.gz) doesn't have to be continuous.

[julia-pfarr]

todo:
add function to check the samples for continuous recordings; if False fill missing rows with continuous timestamps and nans for coordinates/pupil

[Remi-Gau]

OK I need to make sure that we have a test that actually make sure we throw a warning or an error if the output file does not have a continuous timestamp

[julia-pfarr]

But I thought we want to take care of it?

Yes the idea would be to pad physio.tsv.gz with rows of n/a for missing time stamps.

This and the last discussions in the meetings sounded to me as if we check during the conversion process if timestamps are continuous and if not we modify the tsv by adding the rows, no? We can still throw a warning letting the user know that we modified their file in this manner but an error would mean we leave it to the user to take care of it themselves and run our converter again afterwards.

[julia-pfarr]

@oesteban is my comment above correct? (fill timestamps column in order to have continuous recordings and coordinate and pupil columns have n/a for those timestamps)

see entry from meeting 2024/10/03 in our HackMD

[oesteban]

is my comment above correct? (fill timestamps column in order to have continuous recordings and coordinate and pupil columns have n/a for those timestamps)

Correct, the physio.tsv.gz file cannot have "holes" (that is, missing rows at the prescribed sampling frequency).

However, the timestamp column IS NOT mandatory. Whether you have an actual value (which, if present, needs to be consistent) or an n/a on that column becomes unimportant (EXCEPT if you have a referencing physioevents.tsv.gz indexed by the timestamp column, in which case the column becomes mandatory and must have a value).

Does that make sense?

[julia-pfarr]

Correct, the physio.tsv.gz file cannot have "holes" (that is, missing rows at the prescribed sampling frequency).

However, the timestamp column IS NOT mandatory. Whether you have an actual value (which, if present, needs to be consistent) or an n/a on that column becomes unimportant (EXCEPT if you have a referencing physioevents.tsv.gz indexed by the timestamp column, in which case the column becomes mandatory and must have a value).

Does that make sense?

Yes, I understand, that makes sense. Thanks for the quick answer!

For the converter, it will always look for timestamps, so I would write a function that does "if timestamps present and not continuous, fill missing rows". It should then change something like

9199339	1030.8	619.9	9425.0
9199340	1030.8	619.9	9425.0
9199341	1032.9	608.6	9432.0
9199344	1035.7	604.4	9435.0

to

9199339	1030.8	619.9	9425.0
9199340	1030.8	619.9	9425.0
9199341	1032.9	608.6	9432.0
9199342	n/a	n/a	n/a
9199343	n/a	n/a	n/a
9199344	1035.7	604.4	9435.0

right?

[oesteban]

Yes. Just please note that this would also be valid

9199339	1030.8	619.9	9425.0
9199340	1030.8	619.9	9425.0
9199341	1032.9	608.6	9432.0
n/a	n/a	n/a	n/a
n/a	n/a	n/a	n/a
9199344	1035.7	604.4	9435.0

if there's no physioevents file indexed by the timestamp.