I don't think this is practical, or even necessarily more efficient.
Versioning will be a pain, what if packages need conflicting versions/features? There's no nice way for Vite and/or NPM to handle the dependency, so you'd have to rely on nuget to actually ship the component library dependency to the backoffice - which feels kind of wrong.
Modern JS libs like ChartJS are supposed to be used with an "import only what you need" approach. If you want to provide a single shared version you have to import everything people might need. If multiple packages use only a subset you may find that importing the same small thing multiple times is better than importing the whole kit and kaboodle once.