I am trying to read large XML files (100,000 rows) into SQL CE 3.5 database file.
At the moment, I am using XmlTextReader to read xml file line by line and SqlCeConnect / SqlCeCommand TableDirect to write array of data extracted from each line into database.
Using the code below, it takes ~213 seconds to read ~25000 row file. From this, it takes 167 seconds for everything except for writing to database withrecord/rs, 57 seconds to call just itemXmlDoc.LoadXml(xmlTextReader.ReadOuterXml()) inside while loop, and 152 seconds to doitemXmlDoc.LoadXml and 13 itemXmlDoc.SelectSingleNode("item/Subfield").InnerText... so it ends up thatitemXmlDoc.SelectSingleNode is the biggest factor that slows everything down.
What can I do to improve this? Maybe there is a better way to split itemXmlDoc into separate values?
(Before I read itemXmlDoc into string array, it looks something like this: <item><Index>121fg12e<Index><Name>John</Name>.........<Notes>John's profile</Notes></item>)
At the moment, I am using XmlTextReader to read xml file line by line and SqlCeConnect / SqlCeCommand TableDirect to write array of data extracted from each line into database.
Using the code below, it takes ~213 seconds to read ~25000 row file. From this, it takes 167 seconds for everything except for writing to database withrecord/rs, 57 seconds to call just itemXmlDoc.LoadXml(xmlTextReader.ReadOuterXml()) inside while loop, and 152 seconds to doitemXmlDoc.LoadXml and 13 itemXmlDoc.SelectSingleNode("item/Subfield").InnerText... so it ends up thatitemXmlDoc.SelectSingleNode is the biggest factor that slows everything down.
What can I do to improve this? Maybe there is a better way to split itemXmlDoc into separate values?
(Before I read itemXmlDoc into string array, it looks something like this: <item><Index>121fg12e<Index><Name>John</Name>.........<Notes>John's profile</Notes></item>)
using (SqlCeConnection cn = new SqlCeConnection(connectionString))
{
if (cn.State == ConnectionState.Closed)
cn.Open();
using (SqlCeCommand cmd = new SqlCeCommand())
{
cmd.Connection = cn;
cmd.CommandText = "item";
cmd.CommandType = CommandType.TableDirect;
using (SqlCeResultSet rs = cmd.ExecuteResultSet(ResultSetOptions.Updatable | ResultSetOptions.Scrollable))
{
SqlCeUpdatableRecord record = rs.CreateRecord();
while (xmlTextReader.Read())
{
if (xmlTextReader.NodeType == XmlNodeType.Element &&
xmlTextReader.LocalName == "item" &&
xmlTextReader.IsStartElement() == true)
{
itemXmlDoc.LoadXml(xmlTextReader.ReadOuterXml());
values[0] = itemXmlDoc.SelectSingleNode("item/Index").InnerText; // 0
values[1] = itemXmlDoc.SelectSingleNode("item/Name").InnerText; // 1
~~~
values[13] = itemXmlDoc.SelectSingleNode("item/Notes").InnerText; // 13
record.SetValue(index, values[index++] == "NULL" ? null : values[index - 1]); // 0
record.SetValue(index, values[index++] == "NULL" ? null : values[index - 1]); // 1
~~~
record.SetValue(index, values[index++] == "NULL" ? null : values[index - 1]); // 13
rs.Insert(resord);
}
}
}
}
}