Server Side Viewstate

Background

ViewState is a feature provided by ASP.Net that stores information for server controls used on a web page. The primary advantage of this feature from a developer perspective is that it keeps the values of controls on a form. If the user forgot a single required field on a form, ASP.Net can put back the other values they had entered rather than forcing the developer to repopulate them. The information is encoded into a string and placed in a hidden form field called “__VIEWSTATE” that is sent to the client with the response (you can see this by using View Source in your browser when you view a / page). ASP.Net by default also calculates a “hash” of the information and adds it to the ViewState, then on a postback it can validate that the data has not been corrupted. That is a basic description of what viewstate is and how it is implemented, this article will focus on where the viewstate information is stored.

The Problem

After converting our web site from ASP to ASP.Net, we began to have problems with viewstate exceptions. We would receive "The View State is invalid for this page and might be corrupted" about 50 times a day. We searched for known causes for this problem and none of them applied to our situation. After several network traces and calls to Microsoft Support, we determined that when network retransmissions occurred the client would sometimes have a corrupt version of the HTML page (we were never able to determine why TCP did not detect the corruption). If the corruption happened to occur inside the viewstate hidden field value, it would cause the exceptions (the recomputed hash on the server did not match the original hash). We also determined that several browsers (such as older WebTV clients) could not handle large hidden form fields. Our solution was to save the viewstate for the pages on the web server using the Session object instead of the HTML result sent to the client.

Benefits of Server Side ViewState

Avoid corruption problems
Avoid issues with browsers that limit the maximum size of hidden fields
Reduced client page size – since the viewstate information is stored on the server it is not sent in the response to the client which saves download time and bandwidth
Disadvantages of Server Side ViewState
Uses server side memory for the in-process session object
Not as obvious to developers how much space is used by viewstate

The Solution

The idea is to override the default handling of viewstate provided by ASP.Net and implement a custom implementation to store the viewstate information on the server. Originally I was not enthusiastic about the idea of rewriting some of the basic functionality provided by ASP.Net, but it was very easy to do and eliminated our problems.

Since the viewstate is specific to each user, we decided to store the information using the Session object. Our servers use in-process session state (stored in web server memory), so we wanted to limit the amount of viewstate we had to store for each user. We cannot simply store the last viewstate for user, however, because a user can visit a page in Internet Explorer, hit the back button to a previous page and push a button (which would require the viewstate for that page). This requires us to support multiple versions of viewstate for each user. The number of page viewstate values to store is a tradeoff between memory and support for pressing the back button a number of times. The example solution stores the last 5 viewstate values for each user. The viewstate values are stored in a circular array using the modulus operator (% in C#). Every time a user requests a new page on the site, the code increments a request number and uses that number as a key to the proper viewstate in the array on the server. The code uses the modulus of the request number and the size of the array to determine the index into the circular array. This starts storing viewstate for the first page (request number 1) in index 1 of the array (1 % 5 = 1), request 2 would use index 2, etc. This continues until the request number hits 5, which wraps back around and uses index position 0. This is a simple way to use a fixed size array efficiently. The code stores the request number in the “__VIEWSTATE” field that was previously used to store the actual viewstate information. The code consults the web.config file to determine if it should use server side or client side viewstate (changes to the web.config will drop the in-process Session object for the user which will drop all current viewstate, make changes to this during off peak times).

Instructions for Using

The first step in implementing the solution is to download the two C# files, below, that implement the server side functionality and add them to your ASP.Net project.

BasePage.cs

This file includes the class BasePage which is used as a parent class for all pages on the site and can be used without modification. The use of a custom base class has many advantages for an ASP.Net site, in this case it is used to override the default viewstate handling provided by System.Web.UI.Page. The two methods that this class overrides are LoadPageStateFromPersistenceMedium() which is called to reload the viewstate on a postback, and SavePageStateToPersistenceMedium() which is called to save the viewstate before sending the results to the client. The override of these methods simply check if server side viewstate is enabled, then call methods to do the work if it is (otherwise they just invoke the original Page class methods). The LoadViewState() method checks for the __VIEWSTATE field and attempts to convert it to a long (which would fail if the viewstate was actually the original encoded string). If it successfully retrieves the value, it uses another class viewStateSvrMgr to retrieve the viewstate from the session. The SaveViewState() method uses the viewStateSvrMgr class to save the viewstate to the session, and writes the hidden form field __VIEWSTATE to the client page with the request number key. If you are using the .Net 1.1 framework, you should change the name used for the hidden field value, which is stored in the constant VIEW_STATE_FIELD_NAME in this file (anything other than __VIEWSTATE).

All web pages for the site must derive from the BasePage class. The default file webForm1/.cs in an ASP.Net project has the original class declaration “public class WebForm1 : System.Web.UI.Page”, this is changed instead to “public class WebForm1 : ServerSideViewstate.BasePage”.


viewStateSvrMgr.cs

This provides the majority of the code required for server side viewstate. A constant at the top of the file VIEW_STATE_NUM_PAGES controls the number of viewstate values kept for each user. This can be adjusted as necessary to conserve memory, but if the user hit the back button more times than this array size and did a postback, the viewstate data would not be present which will affect page behavior for controls and events. This file also uses a technique that stores an object in session rather than storing multiple independent session variables. The GetViewStateSvrMgr() method checks if an object of this class has already been created and stored in session or if it needs to do so. Then the class can simply store local variables that are all stored in that single session object, which helps avoid pollution of the Session with lots of independent keys with no relation between them. Pages that need to use the services of the class call GetViewStateSvrMgr() and get an instance of the class they can use to get/save viewstate information. The property ServerSideEnabled abstracts the check for whether the server side viewstate is enabled or not (via the web.config setting). The methods SaveViewState() and GetViewState() manage the circular array which is a member of the class, and hence is stored in the object that is placed in the session object for this user.

The last step is to add the following to the web.config file (place just above the </configuration> entry:

<appSettings>
<add key="ServerSideViewState" value="true" />
</appSettings>


Testing

Once the files have been added to the project, the pages have been changed to derive from the BasePage class, and the changes to web.config have been made the site will start to use the session to store viewstate. This can be validated by visiting one of the / pages on the site and using the View Source option of the browser, search for the __VIEWSTATE and notice that it is no longer a large encoded string, but a request number that is the key to the location of the viewstate for the page on the server.

Conclusion

This solution still supports the EnableViewState property for pages and controls and the viewstate size can be seen with tracing, this technique simply changes where the viewstate will be stored. The original driver of using Server Side ViewState was some corruption problems and lack of browser support for large hidden form fields. The use of the described technique solves both of those problems, while also reducing the page size sent to the client.

ServerSideFiles.zip (2.44 kb)

NVARCHAR versus VARCHAR

SQL Server provides both datatypes to store character information. For the most part the two datatypes are identical in how you would work with them within SQL Server or from an application. The difference is that nvarchar is used to store unicode data, which is used to store multilingual data in your database tables. Other languages have an extended set of character codes that need to be saved and this datatype allows for this extension. If your database will not be storing multilingual data you should use the varchar datatype instead. The reason for this is that nvarchar takes twice as much space as varchar, this is because of the need to store the extended character codes for other languages

HTML Editor

I’ve been doing a little research on the use of an HTML Editor for development of internal project, using ASP.NET

Aloha Editor (http://www.aloha-editor.org)

This HTML editor is the bees knees, take a look at the website and check it out, it fantastic

alohaeditor-0.23.9.zip (11.30 mb)

FCKeditor (http://www.fckeditor.net/)

The editor is distributed under the GPL, LGPL and MPL open source licenses. This triple licensing model avoids incompatibility with other open source licenses, making it possible to integrate FCKeditor whenever you want.

FreeTextBox (http://freetextbox.com/default/)

FreeTextBox is the most-used HTML editor for ASP.NET. It is compatible with IE on the PC, and Mozilla and Firefox on all platforms. It is used in major Open Source projects such as Community Server and DotNetNuke as well as excellent packages like Smarter Mail.
Buy FreeTextBox
To purchase a FreeTextBox Professional license or Distribution License with Source code, please click on one of the “Purchase” buttons below. Currently, PayPal is our payment provider. Once you click a “buy” you will be redirected to paypal where you can pay using your PayPal account or a credit card.
Professional License – $49.99
The Professional License allows FreeTextBox to be used in all applications you or your company creates on as many servers as you host, but the license does not include rights to re-distribute FreeTextBox 3.0 Professional features in applications you or your company sell. If you sell or distribute products with FreeTextBox 3.0 Professional features, you must purchase a Distribution license (see below). Upon purchase, you will receive a unique FreeTextBox.lic file which will unlock the Professional features of FreeTextBox 3.0. Professional License Agreement

TinyMCE (http://tinymce.moxiecode.com/index.php)

Open Source under LGPL

CuteEditor (http://cutesoft.net/ASP.NET+WYSIWYG+Editor/default/)

http://cutesoft.net/How+to+Buy/Licensing+Info/default/
This can be purchased from
http://cutesoft.net/ASP.NET+WYSIWYG+Editor/Purchase+CuteEditor+for+.NET/default/

RadEditor (http://www.telerik.com/products/aspnet-ajax/controls/editor/overview/)

RadEditor for ASP.NET AJAX is the leading WYSIWYG web editor. It can replace any TextBox with an intuitive Word®-like editor, which enables even non-technical users to visually manage HTML content. Also available for SharePoint 2007 (MOSS) and DotNetNuke.
Cost $999

On my research I could not find a BSD licensed HTML Editor.

The ones I have used before are
    RadEditor, very good, also very good support and backup.
    TinyMCE, very rich in features and easy to use.
    FCKeditor, little harder to use, and now with extra licences to allow for more general use
   

CSV file parser and writer in C# (Part 3)

This is the last part of an article series exploring reading and writing CSV files with C#/.NET.

Part 1 covered converting the contents of a DataTable into CSV format. Part 2 explained how to read a CSV file back into a DataTable.

Finally we will look at some test cases for use with NUnit, a very important tool that I'll describe in more detail in an upcoming article. There is quite a bit of code, but it is all very simple and easy to understand. Each method marked with [Test] will be called by NUnit and run one of the CSV related methods in turn. Assert statements will then check if the returned values are valid or not.

First, let's check the parser methods. There are several tests here that all test something different; they all pass a CSV formatted string to the CsvParser.Parse method and check the returned DataTable.

using System;
using System.Data;
using System.IO;
using NUnit.Framework;

namespace CsvParser
{
        [TestFixture]
        public class TestCsvParser
        {
                [Test]
                public void UnquotedLine()
                {
                        DataTable table = CsvParser.Parse("one,two,three");
                        Assert.IsNotNull(table);
                        Assert.AreEqual(3, table.Columns.Count);
                        Assert.AreEqual(1, table.Rows.Count);
                        Assert.AreEqual("one", table.Rows[0][0]);
                        Assert.AreEqual("two", table.Rows[0][1]);
                        Assert.AreEqual("three", table.Rows[0][2]);
                }

                [Test]
                public void QuotedLine()
                {
                        DataTable table = CsvParser.Parse("\"one\",\"two\",\"three\"");
                        Assert.IsNotNull(table);
                        Assert.AreEqual(3, table.Columns.Count);
                        Assert.AreEqual(1, table.Rows.Count);
                        Assert.AreEqual("one", table.Rows[0][0]);
                        Assert.AreEqual("two", table.Rows[0][1]);
                        Assert.AreEqual("three", table.Rows[0][2]);
                }

                [Test]
                public void TwoLines()
                {
                        DataTable table = CsvParser.Parse("one,two,three\nfour,five,six\n");
                        Assert.IsNotNull(table);
                        Assert.AreEqual(3, table.Columns.Count);
                        Assert.AreEqual(2, table.Rows.Count);
                        Assert.AreEqual("one", table.Rows[0][0]);
                        Assert.AreEqual("two", table.Rows[0][1]);
                        Assert.AreEqual("three", table.Rows[0][2]);
                        Assert.AreEqual("four", table.Rows[1][0]);
                        Assert.AreEqual("five", table.Rows[1][1]);
                        Assert.AreEqual("six", table.Rows[1][2]);
                }

                [Test]
                public void QuotedMultilineValue()
                {
                        DataTable table = CsvParser.Parse(
                                "\"one\n\"\"beer\"\"\",\"\"\"two\"\"\nbeers\",\"three\nbeers\"");
                        Assert.IsNotNull(table);
                        Assert.AreEqual(3, table.Columns.Count);
                        Assert.AreEqual(1, table.Rows.Count);
                        Assert.AreEqual("one\n\"beer\"", table.Rows[0][0]);
                        Assert.AreEqual("\"two\"\nbeers", table.Rows[0][1]);
                        Assert.AreEqual("three\nbeers", table.Rows[0][2]);
                }

                [Test]
                public void Headers()
                {
                        DataTable table = CsvParser.Parse(
                                "First,Last,Email\nAndreas,Knab,knabar@yahoo.com", true);
                        Assert.IsNotNull(table);
                        Assert.AreEqual(3, table.Columns.Count);
                        Assert.AreEqual(1, table.Rows.Count);
                        Assert.AreEqual("Andreas", table.Rows[0]["First"]);
                        Assert.AreEqual("Knab", table.Rows[0]["Last"]);
                        Assert.AreEqual("knabar@yahoo.com", table.Rows[0]["Email"]);
                }

                [Test]
                public void TrailingNewlines()
                {
                        DataTable table = CsvParser.Parse("test\n\n\n\n\n\n");
                        Assert.IsNotNull(table);
                        Assert.AreEqual(1, table.Columns.Count);
                        Assert.AreEqual(6, table.Rows.Count);
                        Assert.AreEqual("test", table.Rows[0][0]);
                }

                [Test]
                public void Newlines()
                {
                        DataTable table = CsvParser.Parse("test1\x0Atest2\x0Dtest3\x0D\x0Atest4");
                        Assert.IsNotNull(table);
                        Assert.AreEqual(1, table.Columns.Count);
                        Assert.AreEqual(4, table.Rows.Count);
                        Assert.AreEqual("test1", table.Rows[0][0]);
                        Assert.AreEqual("test2", table.Rows[1][0]);
                        Assert.AreEqual("test3", table.Rows[2][0]);
                        Assert.AreEqual("test4", table.Rows[3][0]);
                }
        }
}

Second, the opposite route – testing the CsvWriter. Since it takes quite a bit of effort to completely populate a DataTable, I'll cheat and create the DataTable using the CsvParser, which has been tested separately and is known to work.

using System;
using System.Data;
using System.IO;
using NUnit.Framework;
using CsvParser;

namespace CsvWriter
{
        [TestFixture]
        public class TestCsvWriter
        {
                [Test]
                public void PlainText()
                {
                        string s = "one,two,three\n";
                        DataTable table = CsvParser.Parse(s);
                        string t = CsvWriter.WriteToString(table, false, false);
                        Assert.AreEqual(s, t);
                }

                [Test]
                public void QuotedText()
                {
                        string s = "\"one\",\"two\",\"three\"\n";
                        DataTable table = CsvParser.Parse(s);
                        string t = CsvWriter.WriteToString(table, false, true);
                        Assert.AreEqual(s, t);
                }

                [Test]
                public void MultiLineText()
                {
                        string s = "\"one\nline\",\"two\nline\",\"three\nline\"\n";
                        DataTable table = CsvParser.Parse(s);
                        string t = CsvWriter.WriteToString(table, false, false);
                        Assert.AreEqual(s, t);
                }

                [Test]
                public void Headers()
                {
                        string s = "First,Last,Email\nAndreas,Knab,knabar@yahoo.com\n";
                        DataTable table = CsvParser.Parse(s, true);
                        string t = CsvWriter.WriteToString(table, true, false);
                        Assert.AreEqual(s, t);
                }
        }
}

And that's it! There are more special cases that could be tested, especially invalid input like non-CSV files, but technically everything can be interpreted as CSV, so the question would be what output to expect.

As I mentioned earlier, NUnit is worth a closer look, so check back for more.

CSV file parser and writer in C# (Part 2)

This is the second part of an article series exploring reading and writing CSV files with C#/.NET.

Part 1 covered converting the contents of a DataTable into CSV format; this part explains reading a CSV file back into a DataTable.

First some namespace imports and the namespace declaration for this project:

using System;
using System.Collections;
using System.Data;
using System.Text;
using System.IO;

namespace CsvParser
{

Like the CsvWriter class, all methods in the CsvParser class are static. There are four parser methods that return a DataTable from either a string or text stream, and expecting a header line in the CSV source or not. Only one method has a real body, all others just adjust their parameter signature. The implementation is simple: the method reads one "row" of the CSV source at a time and populates a row in the DataTable. The real meat is the private class CsvStream explained below. There is one utility method that returns an unused column name for a DataTable, in case there are no headers in the CSV source or the headers are not unique.

        public class CsvParser
        {
                public static DataTable Parse(string data, bool headers)
                {
                        return Parse(new StringReader(data), headers);
                }
               
                public static DataTable Parse(string data)
                {
                        return Parse(new StringReader(data));
                }

                public static DataTable Parse(TextReader stream)
                {
                        return Parse(stream, false);
                }

                public static DataTable Parse(TextReader stream, bool headers)
                {
                        DataTable table = new DataTable();
                        CsvStream csv = new CsvStream(stream);
                        string[] row = csv.GetNextRow();
                        if (row == null)
                                return null;
                        if (headers)
                        {
                                foreach (string header in row)
                                {
                                        if (header != null && header.Length > 0 && !table.Columns.Contains(header))
                                                table.Columns.Add(header, typeof(string));
                                        else
                                                table.Columns.Add(GetNextColumnHeader(table), typeof(string));
                                }
                                row = csv.GetNextRow();
                        }
                        while (row != null)
                        {
                                while (row.Length > table.Columns.Count)
                                        table.Columns.Add(GetNextColumnHeader(table), typeof(string));
                                table.Rows.Add(row);
                                row = csv.GetNextRow();
                        }
                        return table;
                }

                private static string GetNextColumnHeader(DataTable table)
                {
                        int c = 1;
                        while (true)
                        {
                                string h = "Column" + c++;
                                if (!table.Columns.Contains(h))
                                        return h;
                        }
                }

The CsvStream class does the actual work – read the CSV source in one character at a time and return meaningful chunks of decoded data, namely data items and rows.

                private class CsvStream
                {
                        private TextReader stream;                 
                       
                        public CsvStream(TextReader s)
                        {
                                stream = s;
                        }

                        public string[] GetNextRow()
                        {
                                ArrayList row = new ArrayList();
                                while (true)
                                {
                                        string item = GetNextItem();
                                        if (item == null)
                                                return row.Count == 0 ? null : (string[])row.ToArray(typeof(string));
                                        row.Add(item);
                                }
                        }

                        private bool EOS = false;
                        private bool EOL = false;

                        private string GetNextItem()
                        {
                                if (EOL)
                                {
                                        // previous item was last in line, start new line
                                        EOL = false;
                                        return null;
                                }

                                bool quoted = false;
                                bool predata = true;
                                bool postdata = false;
                                StringBuilder item = new StringBuilder();
                               
                                while (true)
                                {
                                        char c = GetNextChar(true);
                                        if (EOS)
                                                return item.Length > 0 ? item.ToString() : null;

                                        if ((postdata || !quoted) && c == ',')
                                                // end of item, return
                                                return item.ToString();
                                       
                                        if ((predata || postdata || !quoted) && (c == '\x0A' || c == '\x0D'))
                                        {
                                                // we are at the end of the line, eat newline characters and exit
                                                EOL = true;
                                                if (c == '\x0D' && GetNextChar(false) == '\x0A')
                                                        // new line sequence is 0D0A
                                                        GetNextChar(true);
                                                return item.ToString();
                                        }

                                        if (predata && c == ' ')
                                                // whitespace preceeding data, discard
                                                continue;

                                        if (predata && c == '"')
                                        {
                                                // quoted data is starting
                                                quoted = true;
                                                predata = false;
                                                continue;
                                        }

                                        if (predata)
                                        {
                                                // data is starting without quotes
                                                predata = false;
                                                item.Append(c);
                                                continue;
                                        }

                                        if (c == '"' && quoted)
                                        {
                                                if (GetNextChar(false) == '"')
                                                        // double quotes within quoted string means add a quote       
                                                        item.Append(GetNextChar(true));
                                                else
                                                        // end-quote reached
                                                        postdata = true;
                                                continue;
                                        }

                                        // all cases covered, character must be data
                                        item.Append(c);
                                }
                        }

                        private char[] buffer = new char[4096];
                        private int pos = 0;
                        private int length = 0;

                        private char GetNextChar(bool eat)
                        {
                                if (pos >= length)
                                {
                                        length = stream.ReadBlock(buffer, 0, buffer.Length);
                                        if (length == 0)
                                        {
                                                EOS = true;
                                                return '\0';
                                        }
                                        pos = 0;
                                }
                                if (eat)
                                        return buffer[pos++];
                                else
                                        return buffer[pos];
                        }
                }
        }
}

And that's about it. In an upcoming part of this article I'll share some NUnit test cases for these classes.

CSV file parser and writer in C# (Part 1)

An issue that comes up quite frequently is how to read and write comma seperated value (CSV) files in C#. Surprisingly the .NET libraries have no built-in support for this, and the usual solution to use an OleDb connection to the CSV file with Microsoft Excel's database driver is convoluted and not cross-platform.

CSV files have a very simple structure (source):

  • Each record is one line (with exceptions)
  • Fields are separated with commas
  • Leading and trailing space-characters adjacent to comma field separators are ignored
  • Fields with embedded commas must be delimited with double-quote characters
  • Fields that contain double quote characters must be surounded by double-quotes, and the embedded double-quotes must each be represented by a pair of consecutive double quotes.
  • A field that contains embedded line-breaks must be surounded by double-quotes
  • Fields with leading or trailing spaces must be delimited with double-quote characters
  • Fields may always be delimited with double quotes
  • The first record in a CSV file may be a header record containing column (field) names

In this article, I'll provide simple, but fully functional code to read and write a CSV file according to these rules. In memory, data will be represented as a DataTable, which makes it easy to process; for storage in the file system or transfer over a network the CSV data will be stored as a String or in a Stream.


First, the easy part: writing a DataTable to a CSV file.

public class CsvWriter
{
        public static string WriteToString(DataTable table, bool header, bool quoteall)
        {
                StringWriter writer = new StringWriter();
                WriteToStream(writer, table, header, quoteall);
                return writer.ToString();
        }

        public static void WriteToStream(TextWriter stream, DataTable table, bool header, bool quoteall)
        {
                if (header)
                {
                        for (int i = 0; i < table.Columns.Count; i++)
                        {
                                WriteItem(stream, table.Columns[i].Caption, quoteall);
                                if (i < table.Columns.Count1)
                                        stream.Write(',');
                                else
                                        stream.Write('\n');
                        }
                }
                foreach (DataRow row in table.Rows)
                {
                        for (int i = 0; i < table.Columns.Count; i++)
                        {
                                WriteItem(stream, row[i], quoteall);
                                if (i < table.Columns.Count1)
                                        stream.Write(',');
                                else
                                        stream.Write('\n');
                        }
                }
        }

        private static void WriteItem(TextWriter stream, object item, bool quoteall)
        {
                if (item == null)
                        return;
                string s = item.ToString();
                if (quoteall || s.IndexOfAny("\",\x0A\x0D".ToCharArray()) > –1)
                        stream.Write("\"" + s.Replace("\"", "\"\"") + "\"");
                else
                        stream.Write(s);
        }
}

The methods are static since the whole conversion is done in one method call, there is no need to create object instances etc.

WriteToString will return the CSV file in a string; it is just a wrapper around the more generic WriteToStream method. Both methods take a DataTable and two boolean flags to indicate if you want to write a header line (which would use the column headers of the DataTable) and if you want to quote all values instead of only values that need to be quoted.

Since CSV files do not work well for binary data, your DataTable should not contain any, although the resulting file would still be valid and could be read back in.

WriteToStream just loops through all rows and columns of the DataTable and writes the individual data items to the output stream.

The WriteItem method finally encodes an individual data item and, if necessary or requested, adds quotes around it.

In upcoming parts of this article series, I'll provide and explain code to read a CSV file back into a DataTable, and how to use NUnit to test everything.