I think this is a known issue.
Can't find the exact issue right now, but I believe the fix is to add
data-swipe-close="false"
to the panels div.
OR
It could be bad markup, provide a link to your code and perhaps someone can help you out with an exact fix.