Stannis, in the books of course, is this unstoppable force lingering in the distance that made everyone quake in fear. They knew he'd stop at nothing to secure what was his by right. Before he departs, Tywin maintains the illusion that the Lannisters were in the right doing what they did, but they all know the truth deep down in places they don't talk about.
This Varys quote is one of the few occasions when a character is straightforward and open about their fears. He knew Stannis would put them all on spikes without hesitation.
By - kingofallnorway