Streaming 100 megabyte csv files with WCF

by MikeHogg 21. February 2012 20:35

WCF is very configurable, but that is also the thing that I like least about it.  Often I have a project, and a need, and I’m in WCF documentation, and researching how to do exactly what I want, and I get my config just right, tweaked and tested and it works perfectly, but I never learn a new pattern or algorithm.  They are just config options.  Here is a case where I had to play with the buffer limits to be able to stream very large files, but there are more than a few options related to buffer size and so you have to know exactly which ones work for your case.

Here is my client endpoint.  You will see I am using NetTcp.  I’m in an intranet environment.  I added settings for TransferMode(Stream), ReceiveTimeout(ten minutes), also SendTimeout, as the client also uploaded large files too, and MaxReceivedMessageSize(250MB)

 <system.serviceModel>
        <bindings>
            <netTcpBinding>
                <binding name="NetTcpBindingEndpoint" closeTimeout="00:01:00"
                    openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"
                    transactionFlow="false" transferMode="StreamedResponse" transactionProtocol="OleTransactions"
                    hostNameComparisonMode="StrongWildcard" listenBacklog="10"
                    maxBufferPoolSize="524288" maxBufferSize="65536" maxConnections="10"
                    maxReceivedMessageSize="250000000">
                    <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
                        maxBytesPerRead="4096" maxNameTableCharCount="16384" />
                    <security mode="Transport">
                        <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
                        <message clientCredentialType="Windows" />
                    </security>
                </binding>
            </netTcpBinding>
            <wsHttpBinding>
                <binding name="WSHttpBinding_AnotherClient
                </binding>
            </wsHttpBinding>
        </bindings>
        <client>  
           <endpoint address="anotherendpoint
            </endpoint>
          <endpoint address="net.tcp://somedevserver:8445/SOME_DATA_SERVICE"
                binding="netTcpBinding" bindingConfiguration="NetTcpBindingEndpoint"
                contract="DataServiceReference.IDataService" name="NetTcpBindingEndpoint">
                <identity>
                    <dns value="localhost" />
                </identity>
            </endpoint>
        </client> 
    </system.serviceModel>

You will see similar options in my service config:

 <system.serviceModel>
        <behaviors>
            <serviceBehaviors>
                <behavior name="WindowsService.DataServiceBehavior">
                    <serviceMetadata httpGetEnabled="false" />
                    <serviceDebug includeExceptionDetailInFaults="false" />
                </behavior>
            </serviceBehaviors>
        </behaviors>
      <bindings>
        <netTcpBinding>
          <binding name="NetTcpBinding" maxReceivedMessageSize="250000000" transferMode="StreamedResponse"
                    receiveTimeout="00:10:00" sendTimeout="00:10:00">
            <security mode="Transport">
              <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign"/>
              <message clientCredentialType="Windows"/>
            </security>
          </binding>
        </netTcpBinding>
      </bindings>
        <services>
            <service behaviorConfiguration="WindowsService.DataServiceBehavior"
                name="SOME_DATA_SERVICE.DataService">
                <endpoint address="" binding="netTcpBinding" bindingConfiguration="NetTcpBinding"
                    name="NetTcpBindingEndpoint" contract="SOME_DATA_SERVICE.IDataService">                  
                    <identity>
                        <dns value="localhost" />
                    </identity>
                </endpoint>
                <endpoint address="mex" binding="mexTcpBinding" bindingConfiguration=""
                    name="MexTcpBindingEndpoint" contract="IMetadataExchange" />
                <host>
                    <baseAddresses>
                        <add baseAddress="net.tcp://localhost:8445/SOME_DATA_SERVICE" />
                    </baseAddresses>
                </host>
              
            </service>
          
        </services>
    </system.serviceModel>
  <!-- added this to see trace file -->
  <!--<system.diagnostics>
    <sources>
      <source name="System.ServiceModel"
              switchValue="Information"
              propagateActivity="true">
        <listeners>
          <add name="ServiceModelTraceListener"
               type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
               initializeData="wcf-traces.svclog"/>
        </listeners>
      </source>
    </sources>
  </system.diagnostics>-->

 

And the c# code to stream:

        private System.IO.MemoryStream GetCSVStream(System.Data.Common.DbDataReader dr)
        {
            Initialize();
            try
            {
                System.IO.MemoryStream ms = new System.IO.MemoryStream();
                System.IO.StreamWriter sw = new System.IO.StreamWriter(ms);
                System.Data.DataTable header = dr.GetSchemaTable();
                foreach (System.Data.DataRow r in header.Rows)
                {
                    string column = r["ColumnName"].ToString();
                    if (column == "ID") column = "id";
                    sw.Write(column);
                    if (header.Rows.IndexOf(r) < header.Rows.Count - 1) sw.Write("|");
                }
                sw.Write(sw.NewLine); sw.Flush();
                while (dr.Read())
                {
                    sw.Write(dr.GetValue(0).ToString());
                    for( int x = 1; x < dr.FieldCount; x++ )
                    {
                        sw.Write("|");
                        sw.Write(dr.GetValue(x).ToString());
                    }
                    sw.Write(sw.NewLine); sw.Flush();
                } 
                ms.Position = 0;
                return ms;
            }
            catch (Exception ex)
            {
                LogError(ex, APPNAME);
                return null;
            }
        } 
and that’s it!

Tags:

WCF

About Mike Hogg

Mike Hogg is a c# developer in Brooklyn.

More Here

Favorite Books

This book had the most influence on my coding style. It drastically changed the way I write code and turned me on to test driven development even if I don't always use it. It made me write clearer, functional-style code using more principles such as DRY, encapsulation, single responsibility, and more. amazon.com

This book opened my eyes to a methodical and systematic approach to upgrading legacy codebases step by step. Incrementally transforming code blocks into testable code before making improvements. amazon.com

More Here