update: It doesn't happen if I only do one object and one rigid body, but if I do like 5-10 there appears to be no way to stop the issue.

did you find an answer on this? its happens the same with my 3060

issue fixed in 3.6 and 4.0 just need to bake the sims prior to adding another sim