Near as I can tell, the problem comes in around line 6213 in jQuery 1.4.2. The connection to prototype is that prototype adds a scrollTo method to every DOM element. The connection to IE is that in IE -- but not in FF, for example -- every DOM element has a .document field. The combination seems to cause the code in that area to duck-type DOM elements in IE as being window-like and therefore to return window dimensions rather than element dimensions.
As a workaround, I changed my code from this:
var width = jQuery('.myclass').width();
to this:
var width = jQuery.css(jQuery('.myclass')[0], 'width');
since that's what the code in that area would be doing if that scrollTo method hadn't been added by prototype. That fixed my problem in IE and continued to work as expected in Firefox.
Note also that since the getWindow() function (line 6173ff) uses the same duck-typing idiom to detect windows, I would expect (though I haven't tried it) that .scrollTop() and .scrollLeft(), which rely on getWindow(), would be problematic in this circumstance (called on DOM object in IE on a page that uses both jQuery 1.4.2 and prototype).