jQm has a "make a build" option where you can not include components you do not plan to use, which will reduce the size of your jQm include.
But to say that styling/css/skinning is a small part of a project is just misleading.
If your project requires cross browser support, you will be spending hours tweaking CSS just to make things work across multiple platforms.
jQm removes the need to do that for the most part, letting you focus on the actual coding, not styling of your app.
Plus for disabling the nav on jQm, Ember's router gives you way more functionality by letting it take care of the navigation/url - i.e. dynamic url's based on data, etc.
I guess it all depends on the size of your application, and what the future growth for it looks like.