Memory leak when creating object from html string?
So I'm creating a crawler that starts at a URL, grabs all links from a page, adds them to a list, and goes back to step 1.
For medium to large sites, it's using so much memory that it crashes the page (I'm using Chrome).
I would like some advice on how to reduce the memory used over the long term. Is the constant memory usage increase normal?