I ran ours on tag for releases, but we had multiple developers and we wanted to have more general testing than what we had for all merge requests.
Our solution as implemented immediately builds the add-in for any new commit on a branch named `beta` from literally anyone in our company who wants to contribute, and we have a staged multi-branch release that's owned by our maintainers who QC contributions on a two-week cycle for `main` branch releases. This introduces instantaneous builds but also a measure of quality control. That said, I wouldn't leave the `beta` branch building on demand in a public repo on the wide internet, that only works because I can tell if someone is trying to be nefarious, who it is, and there'd be repercussions for doing so.
We also include functionality that automatically adds any `Function()` defined in a namespace in the `importables` directory into the JMP Scripting Index for JMP 14+, parsing out details of the function to form the documentation in the index. See the video, PDF, or ZIP from my talk for details! Also, please feel free to adapt or adopt anything from our codebase. We're fans of open practices!