MVP

[ericvergnaud]

@antlr/antlr5 ,

Gents,

there seems to be consensus re what the end product could look like, however the required tech stack is not available yet:

• as of writing, WasmGC generation is only supported by Kotlin KMP and J2Wasm (no in-browser generation)
• as of writing, WasmGC execution is only supported by V8 i.e. Chrome and Deno (Node to come in April)

The purpose of this discussion is to outline what our MVP will be. It is a workable step towards our target, but also needs to be a working product, even if it only supports one language.

Tool-wise, since the JVM/GraalVM cannot yet execute the GC in WasmGC, we only have 2 options:

1. for now, keep the tool and add-ons as desktop only - JVM required
2. compile the tool and add-ons to WASM, run them using Deno, only use desktop to compile the generated code to WASM

We need to decide on this. I personally believe we could switch to 2 post-MVP.

Runtime-wise, since the currently available execution platforms are V8 based, the only runtime for the MVP will be TypeScript.

Step-by-step, and assuming we postpone the WASM tool, here is how things would look like:

• keep the tool in Java
• write the Typescript add-on in Kotlin (making it easier to switch to Wasm later)
• migrate the runtime to Kotlin, or use Strumenta's runtime
• compile the runtime to Wasm using Kotlin KMP
• generate the parsers in Kotlin
• compile the generated parsers to Wasm using Kotlin KMP
• generate the parser wrappers in TypeScript
• run the parsers in Deno and/or Chrome

Feature-wise, here is what I'm thinking:

• annotations a.k.a. token categories
• clarified imports/includes
• support for templates
• token value comparators
• parser fragments

Some of these still need to be refined, but that should not prevent us from starting to build.

Awaiting your comments to amend and/or get cracking!

[KvanTTT]

I've just finished ANTLR grammar lexer written in Kotlin: KvanTTT/antlr4#9 All tests are passed (and add new ones). The final code size is comparible to the size of the old ANTLR3 lexer's grammar. If compare to the generated ANTLR lexer, it's much smaller. Also, I'm sure it's more correct and brief. Feel free to use it.

The next step includes writing ANTLR grammar parser in Kotlin but it's more complicated.

I'm doing these things according to my strategy suggested in #24

Also, I wanted to figure out how Kotlin works alongside with Java and it works like a charm!

[ftomassetti]

Also, I wanted to figure out how Kotlin works alongside with Java and it works like a charm!

Great!

[bashor]

as of writing, WasmGC generation is only supported by Kotlin KMP and J2Wasm (no browser)

What do you mean by "no browser"? WasmGC is on by default in Chrome (Chromium-based browsers) and Firefox. Work in Safari is in-progress, a few days ago they merged 2 PRs, and now the nightly build passes over 99% of Kotlin/Wasm compiler tests.

[bashor]

An example: https://play.data2viz.io/sketches/wYqPKL/edit/

[bashor]

Deno is also good, we just don't have deno integration in our tooling out of the box.

[ericvergnaud]

Antlr is a code generator. We want antlr5 to fully run in the browser, such that users can edit their grammar, generate their parser, compile it to WASM and run it entirely in the browser. See http://lab.antlr.org.

[bashor]

If you don't want to rely on deno installed locally, you can find the required infrastructure code in this project https://github.com/Kotlin/kotlin-wasm-examples/tree/main/wasi-example

[ftomassetti]

My concern about Deno is based on my very limited experience in the context of this project: https://github.com/LionWeb-io/lionweb-typescript. The project was initially developed for Deno, but it was extremely difficult to have the same code work in the browser, so eventually the project was refactored to run on Node instead. However @ericvergnaud seems to indicate that having code running both on Deno and the browser is doable, so if that is the case I am less concerned about using Deno

[kaby76]

generate the parsers in Kotlin

StringTemplate is currently used to render the source code for the lexer and parser from a template plus string attributes, If StringTemplate is being dropped (see the comments here and here), what technology are you using in its place? printf()!? Related is trgen, which generates functioning programs from a grammar from the command line. The Antlr4 Tool generates only a tiny part of a parser program (driver code, base class code for recognizers, build scripts, etc). It uses StringTemplate.

[ericvergnaud]

That part is going to change massively, because the language target will no longer be in charge of generating the parser.
It will only need to generate the wrapper. The contract between the tool and the target needs to be specified, but the add-on will be free to use whatever technology they fancy.

[ericvergnaud]

That said the generation of the parser (Kotlin code) by the tool could stick to StringTemplate for now. What would you suggest ?

[kaby76]

I just was wondering what the plan is with regards to StringTemplate. It seems very reasonable to drop StringTemplate because it is not being maintained and draws in old Antlr3 code. Unfortunately, I'm not an expert in template processors, so I don't know what would be the best technology to choose as a replacement. https://en.wikipedia.org/wiki/Comparison_of_web_template_engines .

[KvanTTT]

I'm still thinking about dropping StringTemplate library. Specifically, I'm thinking about introducing a some simple IR (intermediate represantation) for parser. At the end stage this language is being translated to target language using printf(). The IR will contain parser-specific intructions. Actually, StringTemplate internally uses such kind of language. But I haven't decided yet (also, Kotlin has its own IR). And yes, this part is very massive unfortunately.

Another solution is about rewriting StringTemplate but I don't like it.

[ftomassetti]

It seems to me that WASM is a new technology and there are things we need to figure out. Based on this assumption, I would tend towards maximizing the speed at which we can prototype and learn, accepting that this could lead to restart with a different approach in a couple of months, based on what we learn in the meantime.

Tool-wise, @ericvergnaud indicates we can start with option 1 and move to option 2 later on. That seems to me the preferable path as it enables us to move quicker.

Regarding the steps:

• I do not understand what "write the Typescript add-on in Kotlin" means.

• Regarding "migrate the runtime to Kotlin, or use Strumenta's runtime". I am biased but I would lean towards the ANTLR Kotlin's runtime. I would not think of it as the "Strumenta's runtime" as there are major contributions from people outside Strumenta. It has been used by some in production, and the project dealt with the very boring details around setting up the project to make Kotlin multi-platform and the maven publishing happy. So it can save us some effort that would not be particularly useful or interesting to replicate. The only potential issue is with the license, as it is released under the Apache License 2.0, so I guess that if we want to change to BSD (like Antlr4), we need to ask all those who made contributions (or remove their contributions). In practice, if we get the two top contributors besides me it should be done quite easily. But if someone has experience dealing with licenses, please correct me

• Regarding "run the parsers in Deno and/or Chrome" if that helps us to move faster, and if later on we can switch to Node, it seems a perfectly good plan to me.

[ericvergnaud]

Dedicated discussions are usually more effective than general ones (and we already have rants for that 😄)

[kaby76]

@kaby76 you probably mixed up and wanted to answer another topic above?

I'm just trying to "push" the conversation to reveal things that need to be refined. More concerning is that nothing in all 31 Discussions so far constitutes a "requirements specification". People can "get cracking" (here too), to flush out the requirements further, but we need a requirements specification document (pdf or doc or txt) to collaborate. That is why I boldfaced the requirement for Antlr5: removing dependencies with a previous version of Antlr above. It seems to be a requirement, but I don't know.

[ericvergnaud]

@kaby76 I kindly disagree with the "requirements specification" wording. A requirement is not a spec. "Get cracking" means getting into a detailed specification and its implementation, not just the latter. But imho we shouldn't require detailed specs to scope the MVP. We may discover during that specification phase that a requirement is not doable, or involves much more changes than anticipated, and subsequently descope it. That's ok with me.

[kaby76]

A requirement is not a spec.

A "requirement" is not a "specification", but a "requirements specification" is an enumeration of "requirements", which can be functional or non-functional. I try to side with standard terminology that one can look up in Wikipedia, which have been developed over the many years of software practice. #25 (comment) is a good start towards an official "requirement spec" in Word or PDF format or even TXT so one can use git to track changes. But, it's (correctly) incomplete because we don't know the details yet.

I'm okay that people "get cracking"--i.e., "prototyping"--as long as we all understand what is going to happen. We've likely all seen how "prototypes" often get turned into the "product" (MVP or "minimum viable product"??? It's not defined, and there is no "wiki" page for the term). My main concern is how a prototype affects the toolchain post parser generator, e.g., Trash. While the parser generator is important, a tool chain equivalent to Trash is as important as the parser generator because: (1) it allows one to query and refactor the parse tree (e.g., perform static analysis of C# code; turn C# code into Java) from the command-line; (2) it generates turn-key parsers from a grammar, complete with build scripts for command-line testing and to eliminate some of the (hundreds and hundreds of time-consuming) basic questions on StackOverflow, grammars-v4, and antlr4, e.g., "I cannot get such and such parser to work"). Running a parser, or even a parser generator, in a browser is a requirement, which is fine. But I can't use a browser, look at pretty pictures of parse trees, or text boxes in the UI in the browser to copy/paste a parse tree for testing. We need a command-line tool that inputs a grammar and test strings in files, to build a parser and test the parse for input files on Github Actions. In other words, I would hope that this should already be a functional requirement placed in a doc, and labeled "Ken needs this", so there is no confusion and afterthoughts. We don't need to write a prototype to already know that it should be a requirement. We also seem to be developing a requirement for the license of the code. A prototype isn't needed to know this is also a requirement. Enumerating the requirements is key for "product acceptance" so that each and every requirement can be tested. But, you're project leader, so I'll defer. If you want, I can help by starting to write a "requirements spec" doc.

[ericvergnaud]

Thanks for the clarification. I guess we're only enumerating in this MVP discussion significant changes from antlr4 to antlr5 whether additions (annotations, templates...) or subtractions (for example having only 1 target language to start with).
The requirement to be able to parse files is obvious and already implemented, hence why it's not listed. I have no issue adding that to the list (but not sure how useful that is ?).
Removing legacy dependencies is imho good engineering practice, but not a requirement per se - it does not define what the product does or how it can be used, hence why it's not listed.
FWIW there are many ways to provide a "requirements specification" i.e. a list of requirements. In a previous life, I implemented a tool that would capture email threads, on the basis that the "specification is the conversation". It worked out pretty well in that context. In ours, I suspect a GH discussion might be good enough, but if any of us feels there is a need to formally write things down in a PDF (and do the formal work) I won't object.

[ericvergnaud]

The only potential issue is with the license

@ftomassetti would you be kind enough to start looking into that ? I wouldn't want us to engage in a path only to restart at some point due to a licensing issue...

[KvanTTT]

Honestly, I'm not sure if we should rely on any old runtime at all. If we are talking about new major version, probably it makes sense to write everything from scratch.

From the other side, if we want to get a first working prototype ASAP, it could be useful. Yes, that's why it's reasonable to initiate discussion.

[ftomassetti]

I think we can get started experimenting with WASM and clarify a few doubts: if 2 months from now we have learned more about the architecture, and we end up rewriting all the code (but perhaps some gradle scripts and some utilities classes that we verify works on all multiplatform targets) I think we be in a better that we would be if we start from scratch.

Regarding the license change we got approval from 16 out of 17 contributors. I am chasing the last one who contributed 6 commits several years ago (mostly to readme and build scripts we have since then rewrote). If I do not hear from him in the coming days I will remove his changes

[ftomassetti]

This is just to confirm that all contributors of ANTLR Kotlin were ok with the adoption of the dual license, so from today ANTLR Kotlin is available also under BSD 3-Clause

[jimidle]

With no other contributing fators, I would use the Apahce 2.0 license rahter than BSD.

[ericvergnaud]

The other contributing factor is that the code inherited from antlr4 is under bsd3, hence why we've aligned on that, thanks to @ftomassetti for making that happen!

[ericvergnaud]

run the parsers in Deno and/or Chrome

Yes the goal is to accelerate. We will obviously support Node and other browsers once they catch up with the WebAssembly specs, but the more I use Deno, the more I see that as a release-only concern. Deno supports TypeScript ootb and that removes so many problems, it's much more productive than Node.

[ericvergnaud]

Thanks for this in-depth analysis. I wouldn’t be comfortable with part of the software using Apache and another part using BSD. You are indeed encouraged to let Kotlin runtime authors know that their code would serve as the basis for antlr5 wasm runtime. This specs repo will go public once we reach consensus on the MVP.

[ftomassetti]

Perfect, then if I can share this it could make sense to have this conversation in a public issue in the ANTLR Kotlin repository. If you instead prefer to have this more confidential I will try reaching authors by email.

In the meantime I got the green light from the main contributor after me

[ericvergnaud]

👍 It's ok to make it a public conversation.

[ftomassetti]

Thank you. This is now being discussed under Strumenta/antlr-kotlin#144 . We got already permission from authors of 70% of commits

[ftomassetti]

We got permissions from all but the contributors of 11 total commits, so at this stage I think we should be able to move forward, either getting the approval from remaining contributors or by removing their contributions

[ericvergnaud]

Great stuff! Thanks @ftomassetti

[kaby76]

I guess we're only enumerating in this MVP discussion significant changes from antlr4 to antlr5.

The requirement to be able to parse files is obvious and already implemented, hence why it's not listed.

Sorry, I keep hearing that you want to run the parser in a browser, but it sounded as though that is the only way this generated code is going to work.

Removing legacy dependencies is imho good engineering practice, but not a requirement per se - it does not define what the product does or how it can be used, hence why it's not listed.

Strictly speaking, you are right. But in the dozens of projects that I was lead or collaborator, we ended up placing implementation requirements in the requirements specification because it was considered a "feature change".

---

I feel as though I'm not helpful in Antlr5 requirements or implementation. Instead, I will focus more on Saxon for parse tree query and refactoring with Trash/Antlr4. There are some questions I have as to whether it will work as replacement for the Xalan XPath2 engine, which I further modified and ported to C# for cross-document query and multiple DOMs (i.e., a partial parse tree). Unfortunately, Xalan does not implement XQuery, and it is no longer being maintained, so it can't work in the long run.

[ericvergnaud]

@kaby76 sorry to hear that. I feel the opposite i.e. you've been key to some important requirements, and will be when we start diving into details. Your above questions re parse tree queries and refactoring are evidence of that.

[ericvergnaud]

@antlr/antlr5 Gents, I've made this repo public.
I've also created the antlr5 repo, and imported the antlr4 code and history into it.
Then I renamed the 'master' branch to 'main'.
I'll be raising issues in the 'antlr5' repo.

[ericvergnaud]

@kaby76 I'm also thinking of creating:

• a grammars-v5 repo, for hosting refactored grammars
• an antlr5-lab repo, forked from antlr4-lab, for the corresponding online editor (running the backend to be sorted out...)
  What do you think ?

[kaby76]

Sounds good. I have to say your antlr-lab implementation was really good.

[ericvergnaud]

Done!

[KvanTTT]

Let's postpone creating grammars-v5 unless we break backward compatibility. Currently I don't want to maintain both repositories especially considering how many grammars we have.

[parrt]

Probably a good idea for you guys to register the new domain: antlr5.org. That will give you more flexibility for creating URLs to a new lab, etc..

[ericvergnaud]

Good idea, done!