I vaguely remembered doing a bunch of tests that would hit this behaviour as part of a PR I put in back in March, and checking back it definitely was showing the 404 'No Template' page then if you had no template assigned, according to the list of tests I did back then. So unless something has changed in a 13 minor release just since then then the only other thing I can think is if this could be a debug vs non-debug mode difference.
https://github.com/umbraco/Umbraco-CMS/pull/15929