Streaming 100 megabyte csv files with WCF

WCF is very configurable, but that is also the thing that I like least about it.  Often I have a project, and a need, and I’m in WCF documentation, and researching how to do exactly what I want, and I get my config just right, tweaked and tested and it works perfectly, but I never learn a new pattern or algorithm.  They are just config options.  Here is a case where I had to play with the buffer limits to be able to stream very large files, but there are more than a few options related to buffer size and so you have to know exactly which ones work for your case.

Here is my client endpoint.  You will see I am using NetTcp.  I’m in an intranet environment.  I added settings for TransferMode(Stream), ReceiveTimeout(ten minutes), also SendTimeout, as the client also uploaded large files too, and MaxReceivedMessageSize(250MB)

 <system.serviceModel>
<bindings>
 <netTcpBinding>
 <binding name="NetTcpBindingEndpoint" closeTimeout="00:01:00"
 openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"
 transactionFlow="false" transferMode="StreamedResponse" transactionProtocol="OleTransactions"
 hostNameComparisonMode="StrongWildcard" listenBacklog="10"
 maxBufferPoolSize="524288" maxBufferSize="65536" maxConnections="10"
 maxReceivedMessageSize="250000000">
 <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
 maxBytesPerRead="4096" maxNameTableCharCount="16384" />
 <security mode="Transport">
 <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
 <message clientCredentialType="Windows" />
 </security>
 </binding>
 </netTcpBinding>
 <wsHttpBinding>
 <binding name="WSHttpBinding_AnotherClient
                </binding>
            </wsHttpBinding>
        </bindings>
        <client>
           <endpoint address="anotherendpoint
            </endpoint>
<endpoint address="net.tcp://somedevserver:8445/SOME_DATA_SERVICE"
 binding="netTcpBinding" bindingConfiguration="NetTcpBindingEndpoint"
 contract="DataServiceReference.IDataService" name="NetTcpBindingEndpoint">
 <identity>
 <dns value="localhost" />
 </identity>
 </endpoint>
</client>
</system.serviceModel>

You will see similar options in my service config:

 <system.serviceModel>
<behaviors>
 <serviceBehaviors>
 <behavior name="WindowsService.DataServiceBehavior">
 <serviceMetadata httpGetEnabled="false" />
 <serviceDebug includeExceptionDetailInFaults="false" />
 </behavior>
 </serviceBehaviors>
</behaviors>
<bindings>
<netTcpBinding>
<binding name="NetTcpBinding" maxReceivedMessageSize="250000000" transferMode="StreamedResponse"
 receiveTimeout="00:10:00" sendTimeout="00:10:00">
 <security mode="Transport">
 <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign"/>
 <message clientCredentialType="Windows"/>
 </security>
</binding>
</netTcpBinding>
</bindings>
<services>
 <service behaviorConfiguration="WindowsService.DataServiceBehavior"
 name="SOME_DATA_SERVICE.DataService">
 <endpoint address="" binding="netTcpBinding" bindingConfiguration="NetTcpBinding"
 name="NetTcpBindingEndpoint" contract="SOME_DATA_SERVICE.IDataService"> 
 <identity>
 <dns value="localhost" />
 </identity>
 </endpoint>
 <endpoint address="mex" binding="mexTcpBinding" bindingConfiguration=""
 name="MexTcpBindingEndpoint" contract="IMetadataExchange" />
 <host>
 <baseAddresses>
 <add baseAddress="net.tcp://localhost:8445/SOME_DATA_SERVICE" />
 </baseAddresses>
 </host>
 
 </service>
 
</services>
</system.serviceModel>
<!-- added this to see trace file -->
<!--<system.diagnostics>
    <sources>
      <source name="System.ServiceModel"
              switchValue="Information"
              propagateActivity="true">
        <listeners>
          <add name="ServiceModelTraceListener"
               type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
               initializeData="wcf-traces.svclog"/>
        </listeners>
      </source>
    </sources>
  </system.diagnostics>-->

 

And the c# code to stream:

private System.IO.MemoryStream GetCSVStream(System.Data.Common.DbDataReader dr)
        {
            Initialize();
 try
            {
                System.IO.MemoryStream ms = new System.IO.MemoryStream();
                System.IO.StreamWriter sw = new System.IO.StreamWriter(ms);
                System.Data.DataTable header = dr.GetSchemaTable();
 foreach (System.Data.DataRow r in header.Rows)
                {
 string column = r["ColumnName"].ToString();
 if (column == "ID") column = "id";
                    sw.Write(column);
 if (header.Rows.IndexOf(r) < header.Rows.Count - 1) sw.Write("|");
                }
                sw.Write(sw.NewLine); sw.Flush();
 while (dr.Read())
                {
                    sw.Write(dr.GetValue(0).ToString());
 for( int x = 1; x < dr.FieldCount; x++ )
                    {
                        sw.Write("|");
                        sw.Write(dr.GetValue(x).ToString());
                    }
                    sw.Write(sw.NewLine); sw.Flush();
                } 
                ms.Position = 0;
 return ms;
            }
 catch (Exception ex)
            {
                LogError(ex, APPNAME);
 return null;
            }
        } 

and that’s it!