I'm trying to read a large txt file using a C# form application using Visual Studio 2008. This particular file I'm trying to read is about 14Gb but could be as big as 25Gb with about 900,000,000 lines. I'm trying to read each line into List<string>. After a few minutes of running it stops due to running out of memory.
I am getting the following message when debugging:
ContextSwitchDeadlock was detected
Message: The CLR has been unable to transition from COM context 0x1bc68170 to COM context 0x1bc683c0 for 60 seconds. The thread that owns the destination context/apartment is most likely either doing a non pumping wait or processing a very long running operation
without pumping Windows messages. This situation generally has a negative performance impact and may even lead to the application becoming non responsive or memory usage accumulating continually over time. To avoid this problem, all single threaded apartment
(STA) threads should use pumping wait primitives (such as CoWaitForMultipleHandles) and routinely pump messages during long running operations.
Here is the code I'm using to read in the file:
List<string> lst = new List<string>();
StreamReader sr = new StreamReader(str);
sr.ReadLine();//discard file header
while (!sr.EndOfStream)
{
lst.Add(sr.ReadLine());
}
sr.Close();
return lst;
I've assumed this is not the most efficient way to read this file. How would one go about getting all of the lines in the file into List<string>? I'm less concerned with how long it might take as long as it can be done without running out of memory. Thanks.