Convention for the naming and calling of pandoc Lua filter files

[rnwst]

When I write Lua filters, I may want to split up the filter code into several Lua files to group functionality, when one file becomes too unwieldy. To use the filter, I can then no longer simply copy just one filter file to ~/.local/share/pandoc/filters and run it via pandoc -L filter-name.lua. I need to copy the whole repository to that directory and call it via pandoc -L filter-name/filter.lua.

This, however, doesn't seem very elegant. For that reason I thought perhaps we could implement the following convention in pandoc: If the Lua filter called does not end in .lua, e.g. pandoc -L filter-name, pandoc would search for a file ~/.local/share/pandoc/filters/filter-name/filter.lua instead (and in the current directory of course). Keen to hear your thoughts on this proposal!

[badumont]

This sounds interesting. Note, however, that you can already achieve this by altering the value of the variable package.path (see the Lua manual) like this: package.path = package.path .. ';' .. PANDOC_STATE.user_data_dir .. '/filters/my-filter/?.lua'

[rnwst]

Thanks @badumont for your comment. I'm not sure how your suggestion would solve the problem though. Perhaps I didn't do a great job explaining the situation, or I misunderstood something.

To clarify, let's say I have a filter repository filter-name with a bunch of Lua files, and one top level filter file filter-name/filter.lua, which loads the other Lua files. If I now copy or clone the repository to ~/.local/share/pandoc/filters/filter-name in order to make the filter available to pandoc, and I want to call this filter, I need to do so via --lua-filter=filter-name/filter.lua.

This is quite lengthy, and it would be nice to be able to simply do --lua-filter=filter-name. To enable this we would require a naming convention for filter files, so that pandoc knows for which file to look inside of the filter directory (and it seems to me filter.lua would be a reasonable choice).

Changing package.path as you have suggested would need to be done inside of the filter, but that is no use if pandoc cannot find the filter file in the first place.

[badumont]

To clarify, let's say I have a filter repository `filter-name` with a bunch of Lua files, and one top level filter file `filter-name/filter.lua`, which loads the other Lua files. If I now copy or clone the repository to `~/.local/share/pandoc/filters/filter-name` in order to make the filter available to pandoc, and I want to call this filter, I need to do so via `--lua-filter=filter-name/filter.lua`.

Actually there are two issues here, a feature request and a problem that can be worked around (and that I believed was implied in your first message). For the feature request, you will have to wait until it is (maybe) implemented. The problem I addressed is that, when other Lua files are required from `filter-name.lua`, the Lua engine embedded in Pandoc cannot find them because it does not search them under the Pandoc user directory. By setting pandoc.path as I suggested, you can set your file `filter-name.lua` juste under `~/.local/share/pandoc/filters` and the dependent files under a subdirectory thereof, so that you can use your filter with `pandoc -L filter-name.lua` and it will find its required files under `~/.local/share/pandoc/filters/filter-name`.

[rnwst]

Thanks for the clarification @badumont.

Regarding the feature request, I'm happy to implement this myself. The purpose of this discussion is to ascertain whether this would be a desirable feature, and if there is appetite to have a corresponding PR merged, or if someone has a better idea.

Regarding how to require the other Lua files, there are a few options. Adding PANDOC_STATE.user_data_dir to package.path is one option, but then the filter only works if it is located in user_data_dir. My personal favorite would probably be redefining the require function so that it temporarily switches the working directory to the location of the filter file, then runs the old require, and then switches the working directory back to what it was before. That could be done with pandoc.system.with_working_directory or similar, but I haven't thought this through yet. However, that is a different matter, and I'd like the present discussion to focus on the 'feature request' part of this.

[jgm]

This is quite lengthy, and it would be nice to be able to simply do --lua-filter=filter-name. To enable this we would require a naming convention for filter files, so that pandoc knows for which file to look inside of the filter directory (and it seems to me filter.lua would be a reasonable choice).

This sounds reasonable to me. @tarleb any thoughts?

[tarleb]

The Lua default when loading packages is to look for package-name/init.lua. I think we could do the same. But using package-name/filter.lua would work, too. The advantage of init.lua is that it's more idiomatic, while filter.lua could make it easier to split filter functions and reusuable module functions, the latter of which could then go into init.lua. That's flexible, but slightly confusing.

I lean towards using init.lua, but either seems fine to me.

[alerque]

The advantage of init.lua is that it's more idiomatic, while filter.lua could make it easier to split filter functions and reusuable module functions, the latter of which could then go into init.lua. That's flexible, but slightly confusing.

I would suggest that layout would be more than slightly confusing. I would much prefer to see the more idiomatic usage of init and reusable module functions can easily be put in another file or subdirectory and also loaded using idiomatic require expressions.

[tarleb]

In the past, I've also experimented with the following methods to make a filter usable as a module:

1. Return two values from a filter/module script: the filter as the first return value, and the module as the second.
2. Combine the filters and module into a single table: the filters get integer keys, and the module functions string keys. Pandoc ignores all non-integer keys when using the return value as a list of filters, and it will work with require as expected.

I'm partial towards the second, it seems to cause the fewest issues.

[rnwst]

Interesting options. Another option might be to conditionally return either the filter or the module depending on the value of a global variable. Anyway, I think there are enough options here to make the init.lua option work while still being able to use the filter repository as a Lua module, should that be desired.

[bpj]

The MoonScript moon runner modifies Lua’s package.searchers and package.path variables so that files ending in .moon will be read, passed to the MoonScript parser and compiled to Lua code which is then loaded as a package if the corresponding file ending in .lua doesn’t exist, which works well, so I think that in comparison Pandoc’s Lua system prepending $USERDATA/filters/?.lua, $USERDATA/filters/?/init.lua $USERDATA/?.lua, and $USERDATA/?/init.lua to package.path would be relatively benign, compared to the wild stuff moonscript.base does to make things Just Work. (Note that I usually rely on this only during development, otherwise compiling my moon files to lua files, not least so that pandoc and pandoc-lua can use them! 😁)

[tarleb]

About require: I think I'd prefer to keep the original function unaltered, but to add a function pandoc.require that works as suggested by @rnwst. My reasoning here it that it seems like good style to not tamper with the standard Lua functions, because external packages might depend on that exact behavior.

The additional benefit of having a pandoc.require would be that we can make it work with UTF-8: The Lua standard library expects filenames in the encoding of the file system. This leads to an "impedance mismatch" between the UTF-8 encoded strings used in pandoc, and the system-specific encoding expected by the standard library functions. Offering UTF-8 alternatives via the pandoc module would be nice.

[bpj]

Here is general info on how to set LUA_PATH and related variables and the init.lua convention.¹

Generally speaking this is The Right Way. That said it would probably be good to have a way to specifically require filter related modules inside filters which doesn't interfere with other Lua programs (not that I have a lot of those! 😁)

Footnotes

1. That link goes to the Internet Archive so it can be slow to load. ↩

[rnwst]

To be clear, I did not suggest to change the behaviour of the require function for all pandoc users, that was just meant to be a bit of code within my Lua filter. That being said, I really like the idea of a pandoc.require function with the mentioned behaviour, because it would significantly reduce boilerplate code.

[bpj]

I went ahead and added $HOME/.local/share/pandoc/?.lua;$HOME/.local/share/pandoc/?/init.luapandoc/; to my LUA_PATH_5_4 env variable right after ./?.lua;./?/init.lua;$HOME/.lualib/?.lua;$HOME/.lualib/?/init.lua; so now a statement like require"filters.my-filter.support" will Do The Right Thing regardless of whether my-filter/ lives in ./filters/ or in ~/.local/share/pandoc/filters/ and what’s more I can move ~/.lualib/pandochelpers/ where I previously have kept my various filter goodies to ~/.local/share/pandoc/ which at least seems to be an improvement so far.

It would be good if Pandoc when encountering a string my-filter as argument to the -L option in the filters: list in default files would look for $DATADIR/filters/my-filter/init.lua if $DATADIR/filters/my-filter.lua doesn’t exist. Truth to tell I haven’t checked if it already does, so apologies if it does already.

[rnwst]

If we go with the init.lua option instead of filter.lua as @tarleb suggested (and this idea seems to have some support), pandoc's behaviour would need to be changed so that for --lua-filter=filter pandoc would first search for a file filter.lua, and if that cannot be found it would search for filter/init.lua, to match the behaviour of Lua's require function when searching for module files. However, this would be backwards incompatible insofar as --lua-filter=filter.lua would no longer work. I don't have an issue with that, but I'm curious what @jgm has to say about this.

On another note, this concept could perhaps be expanded further by letting pandoc search the entire package.path instead of just the current directory and user_data_dir/filters for filter modules, which would facilitate installation and distribution of filters with luarocks, but I'm not sure about this.

[rnwst]

It would also be backwards incompatible in the sense that running --lua-filter=filter.lua would no longer work, and one would have to run --lua-filter=filter instead.

[tarleb]

I think that giving a file path, whether absolute or relative, should always be possible. It would be very unexpected to users if that was to stop working all of a sudden.

[jgm]

I don't see why --lua-filter=filter.lua would no longer work with this proposal?

[rnwst]

Because --lua-filter=filter.lua would result in a search for filter.lua.lua or filter.lua/init.lua, or if we make it behave exactly like the require function, filter/lua.lua or filter/lua/init.lua. That being said, I agree with Albert that this would be very unexpected for users and should therefore be avoided.

On another note, I'm really starting to like the idea of treating the supplied string like a module name that is required, resulting in a search of all entries of package.path, because it would facilitate installation and distribution of filters via luarocks. We could modify package.path so that it includes user_data_dir, and if the user would like other paths searched as well, they could amend package.path in the Lua interpreter initialisation. I don't know how feasible this is technically though. To avoid breaking the current behaviour, perhaps a simple heuristic would be in order: If the supplied filter ends in .lua it is treated as a path, otherwise as a Lua module name that is required. I need to look at the code to see if this could be implemented cleanly.

[jgm]

I probably got lost in some of the suggestions. The simple model I had was that with --lua-filter PATH pandoc would look first for ./PATH, then for DATADIR/filters/PATH. If neither was found, it would look for ./PATH/init.lua and then DATADIR/filters/PATH/init.lua.

[jdutant]

FWIW, I've encountered the same issue (preference to split a filter in multiple source files). My solution has been to use Lua code combine to write a filter as multiple files, but distribute it in a single file. See e.g. this filter. This solution works out of the box with Quarto as well as Pandoc, and is probably easier for other users if you share your filter.

I wrote a fork of Tarleb's lua filter template that handles creating a single Lua filter from multiple source files. It's not documented yet, but the comments in the Makefile should clarify how it works. Whenever you run make test or make generate or make update, the Makefile uses lua-cc to recreate a single filter file in your main folder and uses it to run tests or generate outputs. You can also run the filter directly from your main source file. The only downside is that you need to match any line numbers in error messages with the original source files.

[rnwst]

Thanks for sharing! I wasn't aware that there are Lua bundlers. That seems to me like more of a temporary workaround rather than a permanent solution. Bundlers are obviously very popular in JavaScript, and with JS there are performance gains to be had by distributing only a single (minified) file as opposed to multiple files, because it minimises the number of requests for your webpage. JS also has source maps, which solve the problem of mapping error messages back to the original code. With Lua filters the situation is different, and I think we should aim to find a solution that does not involve bundling.

Off topic, but I couldn't find your LuaCATS pandoc type definitions, which you seem to be using in the filter you linked. The reason I'm asking is because I recently wrote my own pandoc lua type definitions.

[jgm]

Yes, I was thinking about this bundler solution too. That makes things simpler on pandoc's end, which is always an advantage. It's quite easy to write your own Lua bundler; I've done that.

[rnwst]

Hmm, now I'm curious as to why it would be so easy to write one's own Lua bundler. @jgm, where is your bundler code? 😄
I still don't like that idea though, because it requires an extra build step and because it makes mapping errors to line numbers harder.

[jdutant]

Indeed, I first wrote a simple lua bundler, which is still used in another largish Lua filter I wrote. Its downside is that because it uses alternatives to legit Lua require commands the unbundled source can't be run on its own. That's why I eventually adopted LuaCC instead. But it works if you need a quick solution.

@rnwst "Off topic, but I couldn't find your LuaCATS pandoc type definitions, which you seem to be using in the filter you linked."

Thanks for this, looks nice! The pandoc types annotations in my code come from using the Quarto VS Code extension that includes Pandoc type definitions for the Lua language server.