For anyone looking to add validation to a property editor, I found this commit from Lee useful:
https://github.com/umbraco/Umbraco.CMS.Backoffice/pull/2094/files
Basically, the property editor needs to extend "UmbFormControlMixin", in my case I use another element inside the property editor. So the property editor basically wraps my underlying element. The underlying element also need to extend UmbFormControlMixin, and the property editor needs to add this element to the form.
In the property editor:
protected firstUpdated() {
this.addFormControlElement(this.shadowRoot!.querySelector('my-wrapped-element')!);
}