How can I stop draggable elements from moving when changing other draggable elements?

How can I stop draggable elements from moving when changing other draggable elements?

So I've got a series of divs that are draggable. After positioning them all where I like them, if I use .remove() to get rid of one, all the ones created after that div will move up by the removed div's height.

How do I force these divs to be positioned independently of their spot in the DOM and changes made before that spot so that removing one doesn't reposition the others?