How to remove a clicked button after it is clicked

I have made this code here: http://codepen.io/anon/pen/EyoXVy what I want to achieve is once I click on a button that button dissapears, (and all the other buttons shift) However with the jquery remove method it removes them all, I've also tried using a style to set the visibility to hidden which removes the specific button however does not shift all the other buttons to fill the gaps