Yes, you should disable Instant cache in your case. However, this is not good:
191 SEF URLs, 1853 Metadata, 2374 Sitemap

I mean with 191 URLs you should have max 191 Metadata and Sitemap, otherwise those records are not valid so try to delete the records which does not have any URL from Metadata and Sitemap.

I deleted all the extra sitemap links and started fresh.

I am still seeing issues with the pages not loading however. Either on or two of my virtuemart items loads on the page, then nothing else. Or the page is blank. I go to tools=> clean cache and delete the com_acesef items and the pages then loads as usual. I have global caching turned off so not even sure why this is loading into cache in the first place.

This all started after updating to 1.5.4