There is an Umbraco appsetting entry called ValidElements but I want to set what TinyMCE call extended_valid_elements in their config. Is there some way in Umbraco settings / config to just extend valid elements?

Extended valid elements(extended_valid_elements) is a configuration option of TinyMCE that is normally used to apply additional specific rules on top of the existing validElements ruleset. However Umbraco currently uses this configuration setting to ensure that core 'Umbraco specific' Rich Text Editor functionality is always allowed, eg inserting Macros. It is therefore recommended not to use the extended_valid_elements option in any custom TinyMCE configuration to avoid breaking core Umbraco functionality.

I see. So do you have a suggestion on how to allow script tag? I see 2 options, either ValidElements:

*[*]

or exactly specify all elements/attributes which feels like huge work to do.

https://docs.umbraco.com/umbraco-cms/reference/configuration/richtexteditorsettings Should be able to copy valid_elements from there and then add yours to it