Quantcast
Channel: Windows Forms General forum
Viewing all articles
Browse latest Browse all 12583

Make reading of data into SQL CE database faster? Please help me improve what I have....

$
0
0
I am trying to read large XML files (100,000 rows) into SQL CE 3.5 database file.

At the moment, I am using XmlTextReader to read xml file line by line and SqlCeConnect / SqlCeCommand TableDirect to write array of data extracted from each line into database.

Using the code below, it takes ~213 seconds to read ~25000 row file. From this,  it takes 167 seconds for everything except for writing to database withrecord/rs, 57 seconds to call just itemXmlDoc.LoadXml(xmlTextReader.ReadOuterXml()) inside while loop, and 152 seconds to doitemXmlDoc.LoadXml and 13 itemXmlDoc.SelectSingleNode("item/Subfield").InnerText... so it ends up thatitemXmlDoc.SelectSingleNode is the biggest factor that slows everything down.

What can I do to improve this? Maybe there is a better way to split itemXmlDoc into separate values?
(Before I read itemXmlDoc into string array, it looks something like this: <item><Index>121fg12e<Index><Name>John</Name>.........<Notes>John's profile</Notes></item>)

using (SqlCeConnection cn = new SqlCeConnection(connectionString))
{
    if (cn.State == ConnectionState.Closed)
        cn.Open();
    using (SqlCeCommand cmd = new SqlCeCommand())
    {
        cmd.Connection = cn;
        cmd.CommandText = "item";
        cmd.CommandType = CommandType.TableDirect;
        using (SqlCeResultSet rs = cmd.ExecuteResultSet(ResultSetOptions.Updatable | ResultSetOptions.Scrollable))
        {
            SqlCeUpdatableRecord record = rs.CreateRecord();
            while (xmlTextReader.Read())
            {
                if (xmlTextReader.NodeType == XmlNodeType.Element &&
                    xmlTextReader.LocalName == "item" &&
                    xmlTextReader.IsStartElement() == true)
                {
                    itemXmlDoc.LoadXml(xmlTextReader.ReadOuterXml());
                    values[0] = itemXmlDoc.SelectSingleNode("item/Index").InnerText; // 0
                    values[1] = itemXmlDoc.SelectSingleNode("item/Name").InnerText; // 1
                    ~~~
                    values[13] = itemXmlDoc.SelectSingleNode("item/Notes").InnerText; // 13

                    record.SetValue(index, values[index++] == "NULL" ? null : values[index - 1]); // 0
                    record.SetValue(index, values[index++] == "NULL" ? null : values[index - 1]); // 1
                    ~~~
                    record.SetValue(index, values[index++] == "NULL" ? null : values[index - 1]); // 13
                    rs.Insert(resord);
                }
            }
        }
    }
}





Viewing all articles
Browse latest Browse all 12583

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>