|
7 | 7 |
|
8 | 8 | There are 4 main organizational concepts used in this repo: |
9 | 9 |
|
10 | | -| Concept | Description | |
11 | | -| -------------- | ------------------------------------------------------------------------------------------------------------------------------- | |
12 | | -| App | A single page app that supports operations to be benchmarked (e.g. a TODO app, `/apps/todo` folder) | |
13 | | -| Benchmark | An HTML file that renders an app and measures a specific operation of that app (e.g. adding a new todo, `/apps/todo/todo.html`) | |
14 | | -| Implementation | An implementation of an app (e.g. Preact with class components or Preact using hooks, `/apps/todo/preact-class`) | |
15 | | -| Dependency | A dependency of an implementation (e.g. latest Preact version, `/dependencies/preact/latest`) | |
16 | | - |
17 | | -### Apps, benchmarks, and implementations |
18 | | - |
19 | | -An app is a collection of benchmarks and implementations in the `<repo root>/apps` directory. For example, the `table-app` directory refers to the application that the `js-framework-benchmark` repo uses to measure the performance of various JavaScript frameworks. |
20 | | - |
21 | | -Inside the `table-app` directory we have implemented a couple different benchmarks that measure different operations or aspects of that app. For example, the `replace1k.html` benchmark measures the time it takes to replace 1,000 rows in the table. The `hydrate1k.html` benchmark measures the time it takes to hydrate 1,000 rows in the table. |
22 | | - |
23 | | -For this app, there are also a couple different implementations of that app that we can measure: |
| 10 | +| Concept | Description | |
| 11 | +| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | |
| 12 | +| App | A single page app that supports operations to be benchmarked (e.g. a TODO app, `/apps/todo` folder) | |
| 13 | +| Benchmark | An HTML file that renders an app and measures various aspects a specific operation of that app (e.g. adding a new todo in `/apps/todo/todo.html`) | |
| 14 | +| Implementation | An implementation of an app (e.g. Preact with class components or Preact using hooks, `/apps/todo/preact-class`) | |
| 15 | +| Dependency | A dependency of an implementation (e.g. latest Preact version, `/dependencies/preact/latest`) | |
| 16 | + |
| 17 | +We define these using a folder convention described below: |
| 18 | + |
| 19 | +```text |
| 20 | +- apps/ |
| 21 | + - todo/ <-- An app |
| 22 | + - preact-class/ <-- An implementation of todo using Preact class components |
| 23 | + - preact-hooks/ <-- ... ... using Preact class components |
| 24 | + - preact-compat/ <-- ... ... using Preact compat |
| 25 | + - ... |
| 26 | + - todo.html <-- A specific benchmark using the todo app |
| 27 | + - ... |
| 28 | + - filter-list/ |
| 29 | +- dependencies/ <-- Collection of dependencies whose version can be overridden |
| 30 | + - preact/ |
| 31 | + - local/ <-- A local version of Preact |
| 32 | + - latest/ <-- The latest version of Preact |
| 33 | + - v8/ <-- A specific version of Preact |
| 34 | + - @preact/signals/ |
| 35 | + - local/ |
| 36 | + - latest/ |
| 37 | + - ... |
| 38 | + - ... |
| 39 | +``` |
24 | 40 |
|
25 | | -- `preact-class` uses Preact class components |
26 | | -- `preact-hooks` uses Preact hooks. |
27 | | -- `preact-compat` uses Preact's compatibility layer |
28 | | -- `preact-signals` implements the same app using `@preact/signals` |
| 41 | +All implementations of an app expose the same API. This requirement allows us to compare the performance of different implementations of the same app. For example, we can compare the performance of a Preact class component implementation of a TODO app with a Preact hooks implementation of the same app. |
29 | 42 |
|
30 | | -The pattern described here is used for all apps in the `apps` directory. The folders inside of an app's directory typically are named after the implementation they contain. HTML files within in the app's directory contain the code to load and measure an operation on implementation of that app. |
| 43 | +All version of a dependency have the same API as best as possible. This constraint allows us to compare the performance of different versions of the same dependency. For example, we can compare the performance of Preact v8.0.0 with Preact v10.0.0. Obviously, some things aren't possible, such as implementing hooks in Preact v8.0.0, but we can still compare the performance of the features & behaviors that are shared between versions. |
31 | 44 |
|
32 | 45 | ### Dependencies |
33 | 46 |
|
34 | 47 | Each app is implemented using a JS framework (namely Preact or one of our ecosystem libraries). It is useful to be able to measure and compare how different versions of a framework perform. For example, we might want to compare how Preact v10.5.0 performs compared to Preact v10.6.0. Or how a change in my local repository of Preact performs compared to the latest version of Preact. |
35 | 48 |
|
36 | 49 | All of the dependencies we support comparing are in the `<repo root>/dependencies` directory. Each dependency has a directory named after the dependency (and scoped dependencies are nested so `@preact/signals` will be in the `dependencies/@preact/signals` folder). Inside that directory are directories for each version of that dependency that we support. For example, the `preact` directory contains a `local` directory that will load code from a local clone of Preact repository. It also contains a `latest` directory that contains the latest version of Preact from NPM. |
37 | 50 |
|
38 | | -Inside these directories are the files that are needed to load that dependency. For example, the `local` directory contains scripts to setup loading from a local Preact repository. The `latest` directory contains a `package.json` file that points to the latest version of Preact on NPM. Inside these "proxy packages" you can implement code to massage over minor API differences that may exist between differences versions. For example, in the Preact versions we expose a `createRoot` API that wraps Preact to normalize over API differences between Preact versions. |
| 51 | +Inside these directories are the files that are needed to load that dependency. For example, the `local` directory contains scripts to setup loading from a local Preact repository. The `latest` directory contains a `package.json` file that points to the latest version of Preact on NPM. Inside these "proxy packages" you can implement code to normalize minor API differences that may exist between differences versions. |
39 | 52 |
|
40 | 53 | ## Benchmark server |
41 | 54 |
|
42 | | -In order to actually run benchmarks, we need a server to serve the benchmark HTML files. We use Vite as this server, customized to support our needs. |
43 | | - |
44 | | -The benchmark server is a web server that serves the benchmark apps and the implementations of those apps. It also serves the dependencies that the implementations use. The benchmark server is responsible for mapping the `impl` and `dep` query parameters to the correct implementation and dependency version. |
45 | | - |
46 | | -> TODO: What should implementations import from? Different versions of a library |
47 | | -> might have different exported APIs so they need to import a package that has |
48 | | -> specific version of API. What if we define a common API that neither package |
49 | | -> supports? We could do `declare module "preact"` in apps, but is it useful to |
50 | | -> duplicate the preact TS types in the benchmark apps? We could install a version |
51 | | -> of the libraries in apps/ to then import the types from? |
52 | | -> |
53 | | -> Perhaps we should just align to the latest API of the library and everything |
54 | | -> else must conform to that API if possible. |
55 | | -
|
56 | | -## Decisions |
57 | | - |
58 | | -- Q: Use Tach's web server or Vite |
59 | | - - Requires a web server with the right middleware either way to support mapping "implementation" import to the correct implementation |
60 | | - - A: Vite |
61 | | - - Pro: supports JSX & TSX out of the box |
62 | | - - Pro: Can customize dev experience more |
63 | | - - e.g. load an initial html page to pick a configuration |
64 | | - - Con: Requires running a separate web server |
65 | | - - Unknown: Does Vite's module resolution handle nested `node_modules` with different package versions? Tach does this by creating temporary npm install directories for each version. |
66 | | - - Tach |
67 | | - - Pro: self-contained off-the-shelf |
68 | | - - Con: requires authoring using browser supported syntax |
69 | | - - Con: Have to mentally map ports to implementation/version |
70 | | -- Q: Should benchmark apps be in the same package as CLI or in separate packages? |
71 | | - - A: Probably a separate package since where to find apps is customizable via the CLI config |
72 | | -- Q: How to extend the available implementations & versions from another repository? |
73 | | - - A: Support a CLI config file extends/overrides what implementations & versions are available |
74 | | -- Q: How do implementations specify default versions & versions specify default implementations? |
75 | | - - A: CLI config can specify a default version for a dependency. Perhaps dependencies won't specify a default implementation anymore |
76 | | -- Q: Where do version override packages live? |
77 | | - - A: Any resolvable npm package (so local or NPM)? |
78 | | - |
79 | | -## CLI ideas |
80 | | - |
81 | | -- https://github.com/preactjs/benchmarks/discussions/1 |
82 | | -- `list` or `ls` command to list available dependency versions, and apps & implementations |
83 | | -- Use `prompts` with `prompts.overrides` to provide an easier CLI experience |
84 | | -- `--config` option to specify server config file for finding implementations & versions |
85 | | -- Run Vite server in CLI process? |
86 | | -- Should there be a "build-all" option to build a GH deployable version of all apps? Would probably be difficult as I'd need to build out versions of each implementation for each version that is deployable |
87 | | - |
88 | | -## Running in other repos |
89 | | - |
90 | | -- Consider exporting a function generate a "dependencies" config from the "@preact/benchmark-deps" package that takes in the paths to the relevant local repos |
91 | | -- Then in each repo, a "prepare" step in the benches package.json script would run the function and write the config to a file in the benches directory. The local config will be ignored by Git and so can be customized with local file paths. |
92 | | - |
93 | | -Or |
94 | | - |
95 | | -TODO: Make decision about git submodule or not |
96 | | - |
97 | | -- Should I just simplify this and use Git submodules? Which allows local branching and editing? And removes the need for a CLI config since we can assume more things about the repo structure. |
98 | | -- ??? But how do I configure local repos for Preact & Signals & render-to-string? |
99 | | - |
100 | | -## Vite server |
101 | | - |
102 | | -- Middleware maps `impl` parameter to the correct implementation `index.js`: |
103 | | - - ```html |
104 | | - <script type="module"> |
105 | | - const implParam = new URLSearchParams(location.search).get("impl"); |
106 | | - const implementation = import("./" + implParam + "/index.js"); |
107 | | - </script> |
108 | | - ``` |
109 | | -- Middleware maps any requests for specified `dep` parameters to the dependency version's directory |
110 | | - |
111 | | -## Folder structure |
112 | | - |
113 | | -```txt |
114 | | -- packages/ |
115 | | - - dependencies/ |
116 | | - - preact/ |
117 | | - - local/ |
118 | | - - master/ |
119 | | - - signals/ |
120 | | - - local/ |
121 | | - - master/ |
122 | | - - ... |
123 | | - - cli/ |
124 | | - - apps/ |
125 | | - - table-app/ |
126 | | - - _shared/ |
127 | | - - preact-class/ |
128 | | - - preact-hooks/ |
129 | | - - preact-compat/ |
130 | | - - ... |
131 | | - - 02_replace1k.html |
132 | | - - 03_update1k10k.html |
133 | | - - movie-app/ ... |
134 | | - - todo/ ... |
135 | | - - filter-list/ ... |
| 55 | +In order to actually run benchmarks, we need a server to serve the benchmark HTML files. We use Vite as this server, customized to support our needs. The primary objective is to: |
| 56 | + |
| 57 | +1. Serve benchmark HTML files |
| 58 | +2. Map the `impl` query parameter to the correct implementation directory |
| 59 | +3. Map the `dep` query parameter to the correct dependency version directory |
| 60 | + |
| 61 | +The URL to the HTML file specifies which implementation and versions of dependencies to use. For example, `http://localhost:5173/apps/table-app/hydrate1k.html?impl=preact-hooks&dep=preact@prev-minor` loads the `hydrate1k.html` benchmark of the `table-app` using the `preact-hooks` implementation and the `prev-minor` version of Preact. |
| 62 | + |
| 63 | +The Vite server reads the query parameters, generates an import map to map the `impl` and `dep` query parameters to the correct directories, and inserts it into the benchmark HTML file. This import map maps certain key imports to implementations and versions of dependencies. The benchmark file imports from `@impl` to get the implementation of the app and implementations should directly import their dependencies (e.g. import Preact from `preact` and `@preact/signals` from `@preact/signals`). The Vite server will handle mapping these to special paths that will cause them to be loaded using the HTML import map and not Vite's default node_modules resolution. |
| 64 | + |
| 65 | +Here is an example request flow: |
| 66 | + |
| 67 | +```mermaid |
| 68 | +sequenceDiagram |
| 69 | + participant Browser |
| 70 | + participant Vite |
| 71 | + Browser->>Vite: GET /apps/table-app/hydrate1k.html<br>?impl=preact-hooks<br>&dep=preact@prev-minor |
| 72 | + Vite->>Vite: Add import map to HTML<br> and rewrite @impl to /@impl |
| 73 | + Vite->>Browser: /apps/table-app/hydrate1k.html |
| 74 | + Browser->>Browser: Map /@impl to /apps/table-app/preact-hooks/index.js |
| 75 | + Browser->>Vite: GET /apps/table-app/preact-hooks/index.js |
| 76 | + Vite->>Vite: Rewrite preact import to /@dep/preact |
| 77 | + Vite->>Browser: /apps/table-app/preact-hooks/index.js |
| 78 | + Browser->>Browser: Map /@dep/preact to /dependencies/preact/prev-minor/index.js |
| 79 | + Browser->>Vite: GET /dependencies/preact/prev-minor/index.js |
| 80 | + Vite->>Browser: /dependencies/preact/prev-minor/index.js |
136 | 81 | ``` |
0 commit comments