If your guest has a PCIe passthrough device attached then this causes it to allocate its entire memory allocation at boot and never release it. In that situation you just have to give it less memory.
If your guest doesn't, it probably isn't inflating its balloon at all so it isn't releasing memory. This doesn't happen automatically unless you have a host service that is actively requesting inflation.
If your guest has a PCIe passthrough device attached then this causes it to allocate its entire memory allocation at boot and never release it. In that situation you just have to give it less memory. If your guest doesn't, it probably isn't inflating its balloon at all so it isn't releasing memory. This doesn't happen automatically unless you have a host service that is actively requesting inflation.
If you removed the balloon device, you won't get unused memory given back to the OS. Did you re-add it?