Is there any online documentation on how to use the umbraco headless api browser explorer within the Headless CMS?

Do you mean Swagger? https://docs.umbraco.com/umbraco-cms/reference/content-delivery-api#swagger I can't find a good post about how to use it. But maybe click through it to see how it reacts?

I just use GraphiQL to explore my structure, but then again I use graphQL. Can still be usefull even though you don't use graphql. You'll find it under Settings -> Headless -> GraphQL Playground

Umbraco itself doesn't have a GraphQL implementation, Umbraco Heartcore does but I don't think that's what was being asked about.