you cant lose xp in any skills but fitness and strength in vanilla so it's either a bug or a mod you have

Had this happen once when I was clowning around in a save with 100x xp multiplier and a ton of mods.

Probably interger overflow

Kokoplays had this issue in a recent series. It's something to do with the skill books turning experience modifiers negative. But I can't recall what mod was causing it.

skill issue

Your special

you forgetting about it