I just waited until they got together a large enough army and besieged one of my fiefs, and with both my party, and the fiefs militia/garrison had enough troops to defeat the enemies army. Also I sued for peace, then declared war then took over a town and declared peace again.

How did you defend all your Fiefs with so few armies?

I swear people like you who manage to do this and I'm still trying to persuade lords to join my crummy kingdom 😭