Trying to prevent browser bog/freezing while parsing a giant string (Seeking strategies to do this more efficiently)
I'm trying to make a js solution that takes a large (3MB test file, 10-20MB max in real use) text file from a server and does some processing on that file to accelerate my work flow.
The current issue I'm trying to solve is that the browser is freezing when I try to parse the file after it has loaded since it is thousands of lines long. (plain text) I don't have a problem with the browser taking a while to process the parsing, I just don't want it to freeze 'solid' while it's processing.
I've got the script loading the text file using a Jquery get and loading the result into a string variable, which seems to be working but again causes a bog for ~2-3s while it 'gets' the file from the server. Perhaps this is an incredibly inefficient way of doing this, I'm not sure but would like to know if there's an obviously better way.
(Also if there is a way I can load the file incrementally that would be good to know too. Or sort of "stream" and buffer the file almost like a music player, but instead of playing the music, parsing the information as it loads.)
The primary issue I have is when I use a while loop to parse the string the browser freezes/bogs to the point of thinking that it is crashing/not responding. (1 to 2minute freezes) So I'm looking for a high level suggestion/strategy to use instead of a code fix, per say.
I could definitely expound on my current solution but currently the process flow is:
1) Load text file into string variable using Jquery.get()
2) Parse file using a while loop that says parse until string length is 0.
3) Update a div.html with the parsed string after processing is complete.
Right now #2 is simply converting a "\n" to a <br /> for formatting reasons. The final solution would involve contextual highlighting/formatting so I may end up dropping each line into it's own string in an array. (Unless this is an incredibly bad idea.)
I tried using the jquery .queue(), delay(), and dequeue() functions to help the parsing take a breather, but basically it only seems to make it take longer to complete and if FireBug is running then firebug asks me if I want to stop the script at each delay.
I've tried pauses from 5ms to 750ms, and I've varied the number of parses between pauses from 100 to 4. Even if I only parse 4 lines, then pause 750, it still seems frozen. I'm probably (hopefully) doing the delay wrong but the main point of this post is to see if I'm going about this all wrong in the first place.
Thank you
The current issue I'm trying to solve is that the browser is freezing when I try to parse the file after it has loaded since it is thousands of lines long. (plain text) I don't have a problem with the browser taking a while to process the parsing, I just don't want it to freeze 'solid' while it's processing.
I've got the script loading the text file using a Jquery get and loading the result into a string variable, which seems to be working but again causes a bog for ~2-3s while it 'gets' the file from the server. Perhaps this is an incredibly inefficient way of doing this, I'm not sure but would like to know if there's an obviously better way.
(Also if there is a way I can load the file incrementally that would be good to know too. Or sort of "stream" and buffer the file almost like a music player, but instead of playing the music, parsing the information as it loads.)
The primary issue I have is when I use a while loop to parse the string the browser freezes/bogs to the point of thinking that it is crashing/not responding. (1 to 2minute freezes) So I'm looking for a high level suggestion/strategy to use instead of a code fix, per say.
I could definitely expound on my current solution but currently the process flow is:
1) Load text file into string variable using Jquery.get()
2) Parse file using a while loop that says parse until string length is 0.
3) Update a div.html with the parsed string after processing is complete.
Right now #2 is simply converting a "\n" to a <br /> for formatting reasons. The final solution would involve contextual highlighting/formatting so I may end up dropping each line into it's own string in an array. (Unless this is an incredibly bad idea.)
I tried using the jquery .queue(), delay(), and dequeue() functions to help the parsing take a breather, but basically it only seems to make it take longer to complete and if FireBug is running then firebug asks me if I want to stop the script at each delay.
I've tried pauses from 5ms to 750ms, and I've varied the number of parses between pauses from 100 to 4. Even if I only parse 4 lines, then pause 750, it still seems frozen. I'm probably (hopefully) doing the delay wrong but the main point of this post is to see if I'm going about this all wrong in the first place.
Thank you
Topic Participants
brianh.neteng
Dave Methvin