a quiclk reading from the documentation of the closest method:
Get the first element that matches the selector, beginning at the current element and progressing up through the DOM tree.
so when you click the button, jquery looks if there is a div on the same level, at your case he there is nothing, then it will move up, to the .window div, and then looks if there is a div , the first match will be test1, and this is why it's removed what ever button you click.
so the solution for you is :