Thursday 15 September 2016

Upload/Download large file through WCF REST Service streaming

Hello Frnds,

There was a requirement related to this and before I started working on this I thought everything would be simple and straight forward. But When I finished with the coding as per my thought, I had a problem regarding uploading a big file (>1 GB) and downloading the file and the problem was:
Problem #1: Getting
Problem #2: File was getting downloaded but browser was not able to open the file in specific file format because he din't have the information about "what type of file it is? (file mimetype)".
It took a little while to solve this problem for me and the fix was very simple. All I had to do is to return the file type (mimetype info) as part of response header and here was the trick, and I thought it could be good information to share with you all.
So Let's do it.

I'll start with WCF REST service definition and this is how it should be:

 [ServiceContract]  
   public interface IFileHandlerService  
   {  
     [OperationContract]  
     [XmlSerializerFormat]  
     [Description("Get the template for window")]  
     [WebInvoke(Method = "POST", UriTemplate = "file?fileName={fileName}")]  
     void UploadFile(string fileName, Stream fileContent);  

     [OperationContract]  
     [XmlSerializerFormat]  
     [Description("Get the template for window")]  
     [WebInvoke(Method = "GET", UriTemplate = "file?fileName={fileName}")]  
     Stream DownloadFile(string fileName);  
   }  

and the here is implementation for above service definition:


public class FileHandlerService : BaseFileHandlerService, IFileHandlerService
    {
        public Stream DownloadFile(string fileName)
        {
            Stream fileStream = GetFileStream(fileName); //Here I am getting the actual file as a stream. You could get the file stream from
            //many sources like i.e. from ftp, UNC shared path, local drive itself etc.
            fileStream.Seek(0, SeekOrigin.Begin);

            //Below three lines are very important as it will have the information about the file for browser.
            //without this information browser won't be able to open the file properly and this is what I was I talking about.
            String headerInfo = "attachment; filename=" + fileName;
            WebOperationContext.Current.OutgoingResponse.Headers["Content-Disposition"] = headerInfo;
            WebOperationContext.Current.OutgoingResponse.ContentType = MimeMapping.GetMimeMapping(fileName);

            return fileStream;
        }


        public void UploadFile(string fileName, Stream fileContent)
        {
           // Here you can have your own implementation to save file in different location i.e. FTP Server using ftp, 
           //shared server through UNC path or in the same server etc.
           UploadFileToLocation(fileName, fileContent)
        }
      
    }




This is about the service implementation and here is endpoint and behavior configuration for web.config I have. All important and crucial configurations settings are highlighted. It's difficult to explain all those settings one by one so I would recommend to go through MSDN and read about it if required.

  <system.web>
    <compilation debug="true" targetFramework="4.5.2" />
    <httpRuntime targetFramework="4.5.2" maxRequestLength="2147483647" executionTimeout= "3600" />
    <httpModules>
      <add name="ApplicationInsightsWebTracking" type="Microsoft.ApplicationInsights.Web.ApplicationInsightsHttpModule, Microsoft.AI.Web" />
    </httpModules>
  </system.web>  
<system.serviceModel>
    <services>
      <service name="P21.FileHandlingService.FileHandlerService">
        <endpoint address="" binding="webHttpBinding" behaviorConfiguration="restfulBehavior" 
                  contract="P21.FileHandlingService.IFileHandlerService" bindingConfiguration="WebHttpBindingConfig" />
        <host>
          <baseAddresses>
            <add baseAddress="http://localhost:81/Service" />
          </baseAddresses>
        </host>
        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
       </service>
    </services>
    <bindings>
      <webHttpBinding>
        <binding name="WebHttpBindingConfig" openTimeout="0:25:00" closeTimeout="0:25:00" sendTimeout="0:25:00" receiveTimeout="0:25:00" 
                 maxBufferPoolSize="2147483647" maxBufferSize="2147483647" maxReceivedMessageSize="5368709120" transferMode="Streamed">
          <readerQuotas maxDepth="2147483647"
                        maxStringContentLength="2147483647"
                        maxArrayLength="2147483647"
                        maxBytesPerRead="2147483647"
                        maxNameTableCharCount="2147483647"/>
          <security mode="None" />
        </binding>
      </webHttpBinding>     
    </bindings>
    <behaviors>
      <endpointBehaviors>
        <behavior name="restfulBehavior">
          <webHttp/>
          <dispatcherSynchronization asynchronousSendEnabled="true"/>
        </behavior>
      </endpointBehaviors>
      <serviceBehaviors>
        <behavior>
          <serviceMetadata httpGetEnabled="true" httpsGetEnabled="true" />
          <serviceDebug includeExceptionDetailInFaults="true" />
        </behavior>
      </serviceBehaviors>
    </behaviors>  
    <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true" />
  </system.serviceModel>
  <system.webServer>
    <security>
      <requestFiltering>
        <!--IIS Settings-->
        <requestLimits maxAllowedContentLength="2147483648" />
      </requestFiltering>
    </security>
    <modules runAllManagedModulesForAllRequests="true">
      <remove name="ApplicationInsightsWebTracking" />
      <add name="ApplicationInsightsWebTracking" type="Microsoft.ApplicationInsights.Web.ApplicationInsightsHttpModule, Microsoft.AI.Web" preCondition="managedHandler" />
    </modules>
    <!--
        To browse web app root directory during debugging, set the value below to true.
        Set to false before deployment to avoid disclosing web app folder information.
      -->
    <directoryBrowse enabled="true" />
    <validation validateIntegratedModeConfiguration="false" />
  </system.webServer>   


These settings you can change/modify based on your needs, but you should know two important things here:
1. httpRuntime maxRequestLength : this is the setting which tells what's the maximum request length to send data. i.e. If I need to send data in chunks of 10MB file then it must set to >10MB. Hence all depends on chunks data size.
2. requestLimits maxAllowedContentLength: this the same settings at IIS server level.
3. binding maxReceivedMessageSize: This must be the maximum possible values of you file size which you want to upload.

Hence these values must be set accordingly.

You can test this service using jquery ajax call or C#.

You can consume this service through jquery ajax call or C#. Here is an example for jquery. But below jquery code doesn't have logic to chunking data. Hence this will fail if you want to send large file (i.e. >30 MB).
<!DOCTYPE html>
<html>
<head>
    <title></title>
    <meta charset="UTF-8" />
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script>
    <script type="text/javascript">
        function UploadFile() {
            // grab your file object from a file input
            fileData = document.getElementById("fileUpload").files[0];

            $.ajax({
                url: 'http://localhost:81/Service1.svc/file?fileName=' + fileData.name,
                type: 'POST',
                data: fileData,
                cache: false,
                dataType: 'json',
                processData: false, // Don't process the files
                contentType: false, // Set content type to false as jQuery will tell the server its a query string request
                success: function (result) {
                    alert('successful..');
                },
                error: function (result) {
                    if (result.statusText === 'OK')
                        alert('successful..');
                    else
                        alert('Failed with reason: ' + result.statusText);
                }
            });

        }

        function DownloadFile(fileName, fileType) {

            var url = 'http://localhost:81/Service1.svc/file?fileName=' + fileName;
            window.location(url);             
        }

        function ListFiles()
        {
            $('#tblfiles').append('');
            $.ajax({
                url: 'http://localhost:81/Service1.svc/files', //wcf rest service which return files details.
                type: 'GET',
                cache: false,
                dataType: 'xml',
                processData: false, // Don't process the files
                contentType: false, // Set content type to false as jQuery will tell the server its a query string request
                success: function (result) {
                    var trHTML = '';
                    var i;
                    var x = result.getElementsByTagName("FileDetails");
                    for (i = 0; i < x.length; i++) {
                        var filename = x[i].childNodes[0].textContent;
                        var filesize = x[i].childNodes[2].textContent;
                        var fileType = x[i].childNodes[1].textContent;
                        trHTML += '<tr><td>' + filename + '</td><td>' + filesize + '</td><td>' + '<a href="#" onclick= "return DownloadFile(\'' + filename + '\',\'' + fileType + '\');">Download</a>' + '</td></tr>';
                    }
                    $('#tblfiles').append(trHTML);
                },
                error: function (result) {
                    alert('Failed with reason: ' + result.statusText);
                }
            });
        }
    </script>
</head>
<body>
    <div>
        <div>
            <input type="file" id="fileUpload" value="" />
            &nbsp;&nbsp;
            <button id="btnUpload" onclick="UploadFile()">
                Upload
            </button>
            <br/>
            <button id="btnList" onclick="ListFiles()">
                List Files
            </button>
        </div>
        <br/>
        <div id="filesdiv">
            <table id="tblfiles" border='1'>
                <tr>
                    <th>FileName</th>
                    <th>Size</th>
                    <th>#</th>
                    <!--<th>#</th>-->
                </tr>
            </table>
        </div>
    </div>
</body>
</html>


Note: You have to chunk the data if it is a large file. In this case above jquery code won't work as it tries to send whole data at one shot.

This is the rendered UI for above html.


What I have not shown in above service definition is "rest service method for getting the file details.". This is not a big deal and can be done easily in the way you want.

Here is the C# example which calls this service to upload a large video file (>1GB) and send data in chunks.
try
            {
                string fileName = "The_Angry_Birds_Movie_2016.mkv";
                HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("http://localhost:81/Service1.svc/file?fileName=" + fileName);
                Stream fileStream = File.OpenRead(@"C:\Personal\Movies\" + fileName);
                HttpWebResponse resp;

                req.Method = "POST";
                req.SendChunked = true;
                req.AllowWriteStreamBuffering = false;
                req.KeepAlive = true;
                req.Timeout = int.MaxValue; //this I did for safe side but it shouldn't be case in production.
                req.ContentType = MimeMapping.GetMimeMapping(fileName);
                // The post message header
                string strFileFormName = "file";
                string strBoundary = "———-" + DateTime.Now.Ticks.ToString("x");
                StringBuilder sb = new StringBuilder();
                sb.Append("–");
                sb.Append(strBoundary);
                sb.Append("\r\n");
                sb.Append("Content - Disposition: form - data; name =\"");
                sb.Append(strFileFormName);
                sb.Append("\"; filename =\"");
                sb.Append(fileName);
                sb.Append("\"");
                sb.Append("\r\n");
                sb.Append("Content - Type: ");
                sb.Append("application / octet - stream");
                sb.Append("\r\n");
                sb.Append("\r\n");

                string strPostHeader = sb.ToString();
                byte[] postHeaderBytes = Encoding.UTF8.GetBytes(strPostHeader);
                long length = postHeaderBytes.Length + fileStream.Length;

                req.ContentLength = length;
                req.AllowWriteStreamBuffering = false;

                
                Stream reqStream = req.GetRequestStream();
                
                // Write the post header
                reqStream.Write(postHeaderBytes, 0, postHeaderBytes.Length);

                // Stream the file contents in small pieces (4096 bytes, max).

                byte[] buffer = new Byte[checked((uint)Math.Min(10485760, (int)fileStream.Length))]; //10 MB
                int bytesRead = 0;
                int count = 0;
                while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
                {
                    count += 10;
                    reqStream.Write(buffer, 0, bytesRead);
                    Console.WriteLine("DataWritten:" + count + "MB");
                }
                fileStream.Close();
                reqStream.Close();
                resp = (HttpWebResponse)req.GetResponse();
                
                Console.WriteLine(resp.StatusCode + ":"+ resp.StatusDescription);
            }
            catch(WebException ex)
            {
                Console.WriteLine(ex.Message);
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
            }
            finally
            {
                Console.ReadLine(); 
            }

Note: Above will handle upload/download for any type of files.

That's all. Thanks for reading. Feel free to provide your valuable feedback.



Thursday 2 June 2016

Dynamic Grid with Dynamic controls for Dynamic data using Telerik KendoUI MVC Grid.

Hi friends,

It was an requirement for one of my project and I had to work hard to achieve this. Hence thought it would be worth sharing with you all.

Problem Statement:
Grid should render data with proper controls (like dropdown for list, datepicker for date, checkbox for bool value etc) to provide edit/add feature in InCell mode (User should be able to edit data within the row).
Below picture will tell you what I mean:




As I said, I used telerik kendoUI ASP.NET MVC controls for this, so here are the prerequisite thing which you need.
1. Kendo.MVC.dll (reference for your project)
2. List of scripts files from KendoUI:
  • jquery.min.js (strongly recommended to use the one which you get from Telerik)
  • kendo.all.min.js
  • kendo.aspnetmvc.min.js 
These js files should be loaded in the same order as it is mentioned here because of dependency. Change in order will throw script error.

3. Use kendo css files for look and feel as per your choice.

These all you will get if you install Telerik KendoUI ASP.NET MVC controls by downloading it from Telerik here.



Here we go for solution:

Because of dynamism of data and controls, your most important task would be, designing the Model for Grid's template and data.

First thing you would need columns metadata information for grid so that you can design grid template for dynamic data and this is what you should have.

FEGridViewModel.cs

 public class FEGridViewModel
    {
        public List<Column> Columns { get; set; }
    }

    public class Column
    {
        public string ColumnName { get; set; }

        public ColumnType ColumnType { get; set; }

        public List<KeyValueData> ListData { get; set; }

        public bool Editable { get; set; }

        public bool Unique { get; set; }
    }

    public class KeyValueData
    {
        public string Name { get; set; }

        public string Value { get; set; }

    }
    public enum ColumnType
    {
        Normal,
        List,
        TrueFalse,
        DateTime
    }


Next we should design the view for grid where we will use this model to design the grid template.

View.cshtml
@model FastEditAPI.Models.FEGridViewModel

@(Html.Kendo().Grid<dynamic>()
.Name("dynamicGrid")
.Columns(columns =>
{
    foreach (var cols in Model.Columns)
    {
        if (cols.ColumnType == FastEditAPI.Models.ColumnType.List)
        {
            columns.ForeignKey(cols.ColumnName, new SelectList(cols.ListData, "Value", "Name", cols.ListData.First().Value)).EditorTemplateName("GridForeignKey")
            .Title(cols.ColumnName);
        }
        else if (cols.ColumnType == FastEditAPI.Models.ColumnType.TrueFalse)
        {
            columns.Template(@<text></text>).ClientTemplate("<input type='checkbox' #= " + cols.ColumnName + " ? checked='checked':'' # class='chkbx' />").Title(cols.ColumnName);
        }
        else if (cols.ColumnType == FastEditAPI.Models.ColumnType.DateTime)
        {
            columns.Bound(cols.ColumnName).Format("{0:dd/MM/yyyy}").EditorTemplateName("Date").Title(cols.ColumnName);
        }
        else
        {
            columns.Bound(cols.ColumnName).EditorTemplateName("String");
        }
    }
})
        .ToolBar(toolbar =>
        {
            toolbar.Save();
        })
        .Editable(editable => editable.Mode(GridEditMode.InCell))
        .Pageable()
        .Navigatable()
        .Sortable()
        .Scrollable()
        .DataSource(dataSource => dataSource
        .Ajax()
        .Batch(true)
        .PageSize(20)
        .ServerOperation(false)
        .Events(events => events.Error("error_handler"))
            .Model(model =>
            {
                foreach (var cols in Model.Columns)
                {
                    if (cols.Unique)
                        model.Id(cols.ColumnName);
                    if (!cols.Editable)
                        model.Field(cols.ColumnName, cols.Type).Editable(false);
                }
            })
            .Read(r => r.Action("feGrid_Read", "FE"))
            .Update(r => r.Action("feGrid_Update", "FE"))
            )



Here we go, Based on the column metadata code here will create controls. i.e. if column metadata says it's a List then code will create a drop down. Except this rest other templates are quite straight forward and the dropdown template can be only achieved here through ForeignKey concept.
 columns.ForeignKey(cols.ColumnName, new SelectList(cols.ListData, "Value", "Name", cols.ListData.First().Value)).EditorTemplateName("GridForeignKey")
            .Title(cols.ColumnName);
This ForeignKey concept requires an editor template which you should have under View=>Shared=>EditorTemplates folder with the name of "GridForeignKey" (this name you can provide with your choice but it must be same as provided in above code (as per your view). This is how GridForeignKey.cshtml should be:


GridForeignKey.cshtml
@model object        
@(
 Html.Kendo().DropDownListFor(m => m)      
        .BindTo((SelectList)ViewData[ViewData.TemplateInfo.GetFullHtmlFieldName("") + "_Data"])
)


Next important and interesting thing is model type for grid data which must be of type 'System.Dynamic.dynamic' so we can create dynamically a class of having same properties as the name of columns there in column metadata.

Now let's see how we are going to write/create dataset for grid with dynamic columns name.
Oops I forgot to tell you that, your main action controller's method which will render your view should return the model (I mean FEGridViewModel).

FEController.cs
public ActionResult Index()
        {
            FEGridViewModel model1 = new FEGridViewModel();
            model1.Columns = new List<Column>()
            {
                new Column() { ColumnName = "id", Type = typeof(String), Editable = false, Unique = true, ColumnType = ColumnType.Normal},
                new Column() { ColumnName = "name", Type = typeof(String), Editable = true, Unique = false, ColumnType = ColumnType.Normal},
                new Column() { ColumnName = "corp_name", Type = typeof(String), Editable = true, Unique = false, ColumnType = ColumnType.Normal},
                new Column() { ColumnName = "cc_currency_id", Type = typeof(String), Editable = true, Unique = false, ColumnType = ColumnType.List,
                    ListData = new List<KeyValueData> {
                        new KeyValueData { Name = "Dollars - USD", Value = "Dollars - USD" },
                        new KeyValueData { Name ="Other", Value ="Other" },
                        new KeyValueData { Name ="¥ yen", Value ="¥ yen" },
                        new KeyValueData { Name ="Zimbabwee Dollar", Value ="Zimbabwee Dollar" },
                        new KeyValueData { Name ="French Franc", Value ="French Franc" },
                    }
                },
                new Column() { ColumnName = "delete_flag", Type = typeof(String), Editable = true, Unique = false, ColumnType = ColumnType.List,
                ListData = new List<KeyValueData> {
                        new KeyValueData { Name = "Delete", Value ="Delete" },
                        new KeyValueData { Name = "Not Delete", Value ="Not Delete" },
                    }
                },
                new Column() { ColumnName = "date", Type = typeof(DateTime), Editable = true, Unique = false, ColumnType = ColumnType.DateTime },
                new Column() { ColumnName = "IsActive", Type = typeof(DateTime), Editable = true, Unique = false, ColumnType = ColumnType.TrueFalse },
            };

            return View(model1);
        }



And finally the grid dynamic data will be returned from the controller method "feGrid_Read" of "FE" controller (or whatever details provide in grid UI for event Read based on your code). This controller method will be called through Ajax call by Telerik. Thanks to Telerik to take care such things and making developer life easy.

Now let's create the data set for grid:

public ActionResult feGrid_Read([DataSourceRequest] DataSourceRequest request)
        {
            List<string> strList = new List<string>() { "Dollars - USD", "Other", "¥ yen", "Zimbabwee Dollar", "French Franc" };
            List<dynamic> flexiList = new List<dynamic>();

                for (var i = 1; i <= 30; i++)
                {
                    string curr = strList[new Random().Next(1, 5)];
                    var flexible = new ExpandoObject() as IDictionary<string, Object>; ;
                    flexible.Add("id", (100 + i).ToString());
                    flexible.Add("name", "Data0" + i.ToString());
                    flexible.Add("corp_name", "T");
                    flexible.Add("cc_currency_id", curr);
                    //flexible.Add("cc_currency", new KeyValueData() { Name = curr , Value = curr });
                    flexible.Add("delete_flag", "Not Delete");
                    flexible.Add("date", DateTime.Now.ToShortDateString());
                    flexible.Add("IsActive", i%2==0? true : false);
                    //flexible.Add("flags", new KeyValueData() { Name = "Not Delete", Value = "Not Delete" });
                    flexiList.Add(flexible);
                }
 DataSourceResult dsresult = (flexiList as IEnumerable<dynamic>).ToDataSourceResult(request);
            var serializer = new JavaScriptSerializer();
            serializer.RegisterConverters(new JavaScriptConverter[] { new ExpandoJSONConverter() });
            var json = serializer.Serialize(dsresult);
            return new MyJsonResult(json);
}


because of dynamic data we need to serialize data to json manually. Hence you will be required below two classes which I have used in above code, "ExpandoJSONConverter" and "MyJsonResult". Thanks to Telerik again for creating these two classes.

ExpandoJSONConverter.cs
 public class ExpandoJSONConverter : JavaScriptConverter
    {
        public override object Deserialize(IDictionary<string, object> dictionary, Type type, JavaScriptSerializer serializer)
        {
            throw new NotImplementedException();
        }
        public override IDictionary<string, object> Serialize(object obj, JavaScriptSerializer serializer)
        {
            var result = new Dictionary<string, object>();
            var dictionary = obj as IDictionary<string, object>;
            foreach (var item in dictionary)
                result.Add(item.Key, item.Value);
            return result;
        }
        public override IEnumerable<Type> SupportedTypes
        {
            get
            {
                return new ReadOnlyCollection<Type>(new Type[] { typeof(System.Dynamic.ExpandoObject) });
            }
        }
    }


MyJsonResult.cs
public class MyJsonResult : ActionResult
    {

        private string stringAsJson;

        public MyJsonResult(string stringAsJson)
        {
            this.stringAsJson = stringAsJson;
        }

        public override void ExecuteResult(ControllerContext context)
        {
            var httpCtx = context.HttpContext;
            httpCtx.Response.ContentType = "application/json";
            httpCtx.Response.Write(stringAsJson);
        }
    }


And we are done.

Thank You for reading. Feel free to provide your valuable feedback.