Adding more functions to pandoc.system to avoid encoding issues

[tarleb]

Pandoc expects almost all strings to be UTF-8 encoded, and this is also the encoding used for strings in the Lua subsystem. However, functions from the Lua standard libraries, like io.open, expect filepaths to be given in the system's encoding. This can lead to subtle bugs in filters that become visible only on certain systems.

For example, io.open(PANDOC_SCRIPT_FILE) can fail with a file not found error on Windows systems with a CJK locale and a path that contains non-ASCII strings, as is often the case with paths that include the username.

We do provide the pandoc.text.toencoding function for filter authors to work around this, but it would be easier to simply use the Haskell runtime system to perform file operations using UTF-8. So I'd propose to add the following UTF-8-aware functions:

• pandoc.system.rename(old, new)
• pandoc.system.remove(filepath)
• pandoc.system.copy(source, target)
• pandoc.system.read_file(filepath)
• pandoc.system.write_file(filepath, content)

We'd then recommend to use the systems module instead of io and os. That should make filters and other pandoc Lua scripts more robust.

CC @cscheid

[tarleb]

I already created the functions in hslua, (changelog), this is just to ask if they should be exposed in pandoc, and whether more functions should be added.

[jgm]

No objections!
Perhaps creating directories is another thing filters would want to do involving file paths.