Build automation: Merging and minifying JavaScript files with Ajax Minifier

In my Library I have come to the point where it is gets harder and harder to maintain it. I' am currently over the 3000 lines, so it got time to split up my library in separate files and divide it in logical groups. After splitting everything up in separate files, I was looking for away to merge all these files into one file and have the option to minify it. This is also a step I wanted to make, so my lib could get more and more mature. The last thing I wanted was to integrate it with visual studio – which is my IDE that I’m using to develop the library – and be portable. With that last thing I mean I didn’t want to install something on my machine to do the magic.

After some searching I choose the Microsoft Ajax Minified. It comes in a nuget package which makes it easy to install. Next to that it integrates smoothly with visual studio using MS Build Tasks. This means it will merge and minify the files every time I build the project. There is only a little disadvantage where I will start with.

Set-up Ajax minifier

To start, the first thing we need to do after installing the nuget package, is adding the following files to your project (I have added a new folder Build to add them in.)

• AjaxMin.dll
• AjaxMinTask.dll
• AjaxMinTask.targets

All these files can be found in the tools/net40 folder in the nuget package folder of the Ajax minifier

<folder where the solution is located>\packages\AjaxMin.<version>\tools\net40

The disadvantage of this way of working is that you need to copy the above files when getting the latest version of the Ajax minifier from nuget.

The last thing you need to do is editing the .csproj file so it will use the added build task. You can do this by unloading the project. (right click on the project –> Unload project)

Once you have done this, you can right click on your project and choose for edit <project name>.csproj

This will open an XML file containing information about the project. At the end of the file before the project closing tag, you must add the following line.

<Import Project="$(MSBuildProjectDirectory)\Build\AjaxMinTask.targets" />

Save this file and right click on the project again and choose for Reload Project. This will load the project again with the modifications.

Ajax minifier manifest File

Once all this is done, we can add an Ajax manifest file. In this file we can declare which files must be merged and minified. Also we declare the location where these files must be saved. Optionally we can provide some arguments to overwrite the default behavior. More information about the arguments can be found here. Note that these arguments are the same as the arguments you can provide when working with the command line tool.

The location of this manifest file doesn’t matter, but the extension needs to be “.ajaxmin”. For my library I have put this file also in the build folder where all the Ajax minifier files are located. You can also provide multiple output tag so you can create for example a minified and a non minified file.

Bellow you can find an example of an Ajax manifest file.

<?xml version="1.0" encoding="utf-8" ?>
<root> 
  <output path="Scripts\output.js">
    <arguments>-clobber -pretty</arguments>
    <input path="file1.js"/>
    <input path="file2.js"/>
  </output>
</root>

The arguments I am providing are “-clobber” to overwrite files even when they are flagged read-only, “-pretty” this provides a nice formatted JavaScript file that is readable. (not minified)

By default the output files will be added to a content folder in your project (must exist or you will get an error.) If you want to determine the path in your manifest file, then you need to change the AjaxMinTask.targets file we added in the first part. This is also an XML-file containing the build task definition.

In this file you will find a tag called “AjaxMinManifestTask”. On that element you have several attributes and the one we need is Output Folder. In my case the value of it is “$(ProjectDir)”. With this I can address the location in my manifest file starting from the root of my project. As said earlier: the only thing you need to keep in mind is that the folder structure must be present.

When all this is done, the only thing you need to do is build the project, and the files will get created.

Conclusion

Working with the Ajax Minifier gives you a lot of options for minifying and merging your JavaScript files. Next to that it can also minify CSS files. And as seen above with some work you can easily integrate it into the build of your project.

Programming in HTML5 with JavaScript and CSS3 Specialist

As of today, I am a Programming in HTML5 with JavaScript and CSS3 Specialist. I took the 70-480 exam and passed with a score of 760. Tough I already have quite some experience with HTML5 and JavaScript, the exam was tougher than I expected. That is why I wanted to make this post so every who is thinking about doing this exam gets a head start. Also every one who wants to do the exam, can do this for free until 31/03/2013. More information about this you can find here.

To start, Microsoft provides a one day online course to prepare for the 70-480. This course can be found on the Microsoft virtual academy. On that site you can find courses for several technologies. These courses are divided into several modules and every module has a survey you can take when you completed the resources provided. This is a good starting point if you want to study for the 70-480, but it is certainly not enough. The course is too brief.

If I look back at the exam, jQuery is very important. In a lot of questions jQuery was used in the code examples and solutions. So make sure you know how jQuery works and how to use it. Also a great part of the questions handled about AJAX request. It is definitely worth to inspect the AJAX implementation of jQuery with all its options and events. I got several questions about it.

A second great part of the questions handled about forms and validations. Make sure you know how regular expressions work. I had at least two questions that handled about choosing the right regex. And to complete the HTML5 element part I got some cases in which I note which HTML5 structure element I should use.

The questions about CSS handled the most about positioning, the box model, display modes, media queries and selectors.

The last part was about JavaScript with the focus on the object oriented aspect and closures. It is also a important to know how event handling works in JavaScript. For the API’s the most important ones are Web workers, local/session storage. It is good to know how they work and how you can use them.

For every one who takes the challenge, I wish you good luck and I hope I provided some useful hints.

IndexedDBViewer: Take a look inside your indexedDB

Some days ago I released a new version of the IndexedDBViewer 1.1.0. The IndexedDBViewer is intended for web developers who want to sneak into their indexedDB database. It allows you to inspect the structure of your database as well as the data stored inside this structure. The difference with the previous version is that it no longer needs the jQueryUI library. This way I eliminated at least one reference. The following references are still necessary:

• jQuery (version 1.8.2 or higher)
• linq2indexedDB (version 1.0.19 or higher)
• indexedDBViewer (1.1.0 – JavaScript + css file)

If you are using nuget, you can get all the resource by searching for the indexedDBViewer.

The second major change is that the viewer can easily be added to an existing page. The only thing you need to do is add a div with “indexedDBViewer” as id and data-dbName attribute to pass the database you want to inspect. The rest will be handled by the script in the viewer

   1: <div id="indexedDBViewer" data-dbName="database name"></div>

Once this is done and you navigate to the page with the viewer, you will get the following result

In the bottom you will see the view appear. On the left pane you get an overview of the database structure. This a list with on top the name of the database. Under that you will find child nodes that represent the object stores present in the database. If we descend an other level we can see the indexes present on the object store. If you click on the “+” or “-“ next to the names, you can expand or hide the structure beneath.

If you click on the database name in the navigation pane, you will get information about the database and it’s structure.

• In the general block you will see the name of the database and the version it is currently in
  
• The object stores block gives you an overview of all the object stores present and how they are configured.
  
• The indexes block shows all the present indexes and how they are configured.

When you click on one of the object store names in the navigation pane, you will get all the data present in the object store. Because the data is saved as a key/value pair, you will see the key with his corresponding value. If the value is an object or contains objects, then you can inspect them by clicking on the “+” to expand and “-” to hide the details.

If you click on one of the index names in the navigation pane, you will get – similar as for object stores – all the data present in the object store. But in this case you will see a little more. Besides the key of the index and the value you will see the key the value has in the object store. This can be found under the “primary” key column.

As last there are some little extra features:

• If you click on the top border of the viewer and drag it up or down, then you can change the height of the viewer.
  
• if you click on the “-” in the right top of the viewer, you can hide the viewer. If you want it to appear again, then you have to click on the “+” on the right bottom of the page.

Conclusion

With this Chrome like indexedDBViewer you can inspect the structure of your database inclusive all data stored within it. This with the advantage that it runs inside the browser, so you can use it cross-browser.

IndexedDB: MultiEntry explained

For a long time I was not sure what the purpose of the multiEntry attribute was. Since non of the browsers supported it yet, but since sometime Firefox and even the latest builds of Chrome support it, it all came clear to me. The multiEntry attribute enables you to filter on the individual values of an array.For this reason, the multiEntry attribute is only useful when the index is put on a property that contains an array as value.

When the multiEntry attribute is on true, there will be a record added for every value in the array. The key of this record will be the value of the array and the value will be the object keeping the array. Because the values in the array are used as key, means that the values inside the array need to be valid keys. This means they can only be of the following types:

• Array
• DOMString
• float
• Date

So far for the theory, an example will make everything clear.

In the example below I will use an object Blog. A blog contains out of the following properties:

   1: var blog = { Id: 1

   2:            , Title: "Blog post"

   3:            , content: "content"

   4:            , tags: ["html5", "indexeddb", "linq2indexeddb"]};

In the indexeddb we have an object store called blog which has an index on the tags property. The index has the multiEntry attribute turned on. If we would insert the object above, we would see the following records in the index:

key	value
“"html5”	{ Id:1, Title: “Blogpost”, content:”content”, tags: [“html5”, “indexeddb”, “linq2indexeddb”]}
“indexeddb”	{ Id:1, Title: “Blogpost”, content:”content”, tags: [“html5”, “indexeddb”, “linq2indexeddb”]}
“linq2indexeddb”	{ Id:1, Title: “Blogpost”, content:”content”, tags: [“html5”, “indexeddb”, “linq2indexeddb”]}

So for every value in the array of the tags attribute, a record is added in the index. This means when you start filtering, it is possible that the same object can be added to the result multiple times. For example if you would filter on all tags greater then “i”, the result would be 2 times the blog object I use in this example.

Promises: jQuery deferred object vs WinJS Promise

When I was adjusting my Linq2IndexedDB library to enable Windows 8 development, I had some little issues porting the jQuery promises to WinJS promises. In this post I will show you some differences I had issues with and how I fixed them.

Passing context

The first problem I ran into: WinJS promises doesn’t support passing a context. Because I don’t take advantage of the context yet, this wasn’t an issue for me yet. I easily solved it by writing a wrapper around the 2 promises. In case of the jQuery promise, the context gets passed. And in case of the WinJS promise, I just ignore it for the moment and hope it will get implemented in the future.

   1: function promiseWrapper(promise) {

   2:     if (isMetroApp) {

   3:         return new WinJS.Promise(function(completed, error, progress){

   4:             promise({

   5:                 complete: function (context, args) {

   6:                     completed(args);

   7:                 },

   8:                 error: function (context, args) {

   9:                     error(args);

  10:                 },

  11:                 progress: function (context, args) {

  12:                     progress(args);

  13:                 }

  14:             });

  15:         });

  16:     } else if (typeof ($) === "function" && $.Deferred) {

  17:         return $.Deferred(function (dfd) {

  18:             promise({

  19:                 complete: function (context, args) {

  20:                     dfd.resolveWith(context, [args]);

  21:                 },

  22:                 error: function (context, args) {

  23:                     dfd.rejectWith(context, [args]);

  24:                 },

  25:                 progress: function (context, args) {

  26:                     dfd.notifyWith(context, [args]);

  27:                 }

  28:             });

  29:         }).promise();

  30:     }

  31: }

Passing multiple parameters

A second problem I had was the fact that the WinJS only allows one argument to be passed when calling a complete, error or progress callback. Because I needed to pass multiple values in my library, I needed to rewrite every complete, error and progress method I called. Instead of just passing multiple arguments to the callback methods, I needed to wrap the arguments into an array so they could be passed as single argument.

But that wasn’t enough. Because the jQuery promise is smart enough to convert an array of arguments into a callback with multiple arguments, I needed to wrap the array of arguments into an other array (If you look in the sample above, you will see in case of the jQuery promise, brackets (‘[]’) were added around the args argument.). I needed to do this, because it was the only way to get the same signature when working with the WinJS & jQuery promise.

   1: promiseWrapper(function (pw) {

   2:     pw.complete(context, [arg1, arg2]);

   3: });

Progress Event Doesn’t fire in Winjs promise

The last issue I suffered was the fact that a progress event in didn’t fire in some cases. After a little investigation, I came to the conclusion that I was calling the progress event, before the promise object got created. I first noticed this when I was creating a transaction on the IndexedDB API. When a transaction was created, I fired a progress event with the transaction data. In other methods, where I needed the transaction, I used this the progress call to execute my queries. Once the transaction was committed, the complete event got fired. Because the progress call never got called, the query was never executed. This way the transaction was immediately committed and the complete callback got called without any action taken.

To fix this I delayed the progress call a little bit. By adding a setTimeout of 1 ms I noticed my problem was solved, and my progress events got called.

   1: if (isMetroApp) {

   2:     setTimeout(function () {

   3:         var txn = db.transaction(objectStoreNames, transactionType); 

   4:         txn.oncomplete = function (e) {

   5:             pw.complete(txn, [txn, e]);

   6:         };   

   7:         pw.progress(txn, [txn]);

   8:     }, 1);

   9: }

Conclusion

As seen above working with the WinJS is a slice different of working with the deferred object in jQuery. But with the given workarounds, it is possible to solve the most issues. I hope that Microsoft will take a look at the jQuery deferred object in the future and add some of the jQuery capabilities (context, multiple arguments) into the WinJS promises. And hopefully the progress bug gets solved, so the ugly setTimeout can disappear in my code.

IndexedDB: Keys: efficiently retrieving data from a database

In order to retrieve data efficiently from an IndexedDB database, each record is organized by its key. The only condition for the key is that it is a valid key. A valid key can be one of the following types:

• Array (Is only valid when every item inside the array is a valid key and the array doesn’t contains it self. Also all non-numeric properties are ignored and will not affect whether the array is a valid key)
• DOMString
• Date (The Primitivevalue (internal property) of the date object can’t be NaN)
• Float (The key can’t be NaN)

When comparing keys, the order in the list above applies. Array is greater than all DOMStrings, DOMString is greater than …

Comparing arrays is done by comparing the values inside the array (as long that both arrays got values on the position). If the values on the same position differ, then the outcome of that comparison will determine which array is greater. If the values on each position are the same, the length of the array will determine the which one is greater or equal if the arrays have the same length.

Complex Keys

The IndexedDB API also support complex keys. This are keys who refer to properties nested inside an object. This way you are able to filter on data that is available are properties of a nested object. In the example below we have an person object. Besides a name and firstname property it contains an address property. This address property is also an object containing properties like city, street, … If we want to filter data on its city we can do this by defining a key like this “address.city”.

   1: var person = {

   2:     name: "name",

   3:     firstname: "firstname",

   4:     address: {

   5:         street: "street",

   6:         city: "city"    

   7:     }

   8: }

Since Linq2IndexedDB 1.0.11 this is also supported for all filters, even for filters that don’t take advantage of the IndexedDB API. For this reason it is advised to use the linq2IndexedDB.prototype.utilities.getPropertyValue method to determine the value by it’s property name. Two arguments need to be provided:

• first argument is the object containing the data
  
• second argument is the propertyName string. For example “address.city”

The return value of the method is the value at this location if it exists, otherwise undefined will be returned.

Note that this nesting only works for objects. If the property contains an array of objects, you won’t be able to use this way to filter your data.

Object store keys

Inside an object store, keys even have a more special reason of existence. In here keys are used to uniquely identify an object. This means you can never add an object twice with the same key, but this enables you to update and delete objects when you know it’s key. For an object store, there are 2 ways you can provide a key:

• Inline key: the key is present inside the object (A KeyPath must be defined on the object store)
  
• External key: the key is provided separately (in most cases the value of the key/value will be a primary type like a string or float)

In both cases you can choose for the option to let the key be generated by the IndexedDB API. You can do this by setting the object store autoIncrement attribute on true when creating the object store. In this case the key will have a numeric value.

Index keys

The keys in for an index are always defined by a KeyPath. These keys can be complex keys as defined in the section “Complex keys” above. Every object in the object store keeping the index that has a value for the key path a record will be added. The key will be the value of the property defined by the key path. For example we have a keyPath “firstname”. In the example below, the key in the index record would be “Kristof” and the value the person object.

   1: var person = {

   2:     name: "Degrave",

   3:     firstname: "Kristof"

   4: }

In the index you can also have 2 special cases. The first one is the unique attribute. This will make sure, that a value for this keyPath can only be defined once in a value of the object store. This constraint will be checked when you add data to an object store. If there is an object present with the same value for that property, an ConstraintError will be thrown.

A second attribute is the multiEntry. This allows to have multiple values for only one key. In stead of adding a new record for every object with the same value, the key is only added once and all the objects that have same value are added to a collection.

Conclusion

Now you should have insight how the keys work inside the IndexedDB API. With the complex keys you will be able to filter on data with nested object, this way you don’t need to provide an object store for every object you want to store. Also the Linq2IndexedDB framework also supports these complex keys and you can even use them in your custom filters.

Linq2IndexedDB: Debugging IndexedDB

In my latest version of the Linq2IndexedDB 1.0.7, I added the viewer object. This object allows you to inspect your IndexedDB database and gives you an overview of all the object stores and indexes that are present. It even gives you details about how they are configured and what data they contain. Also using this, you can take a look at your database in various debugging tools like:

• Google Chrome Developer tools
• Firefox Developer tools
• IE Developer tools
• Visual Studio

The viewer object contains 4 properties:

1. configuration: In here you will find the way the Linq2IndexedDB framework is configured. (The same properties as you can define in the configuration object are present +  a property indicating if framework is running in auto create mode.)
2. name: The name of the IndexedDB database
3. objectStores: A collection of all the object stores that are present in the database
4. version: The current version of the database

Inside an object store object, we can see the following:

1. autoIncrement: indicates if the keys for the object store are generated by the IndexedDB API
2. data: A collection of all the data available in the object store. These data objects are shown as key/value pair objects.
3. indexes: A collection of all the indexes available on the object store.
4. keyPath: The property inside the object that keeps the key
5. name: The name of the object store

The last level we can take a look at, are the indexes. The following information will be provided:

1. data: A collection of all the data available in the object store. These data objects are shown as key/value pair objects.
2. keyPath: The property inside the object that keeps the key
3. multiEntry: Determines if a key is only once defined in the index (which means the value keeps an array of all the values in the object store who match) or the key can be present multiple times (once for every value that matches in the object store.)
4. name: The name of the index

conclusion

I hope this debugging information can help you speed up your software development and finding bugs while working with the IndexedDB API. The latest version of the framework can be found on codeplex, nuget and MyGet.

Update 28/07/2012

I added a new version of the framework, because there were some little bugs with the viewer. The following things change:

• The viewer no longer blocks the deletion or upgrade of the database
• The viewer now gets updated when the structure of the database changes
• The viewer is default disabled for performance issues. If you want to enable it, add true as 3th argument on the function to create a linq2indexeddb function

   1: var db = $.linq2indexedDB("test", null, true);

Linq2IndexedDB: Custom filters

Since the 1.0.5 version of the Linq2IndexedDB framework, I added support for custom filters. And since the 1.0.6 version (released last week) you can add a function to the where clause to filter your data. These functionalities are provided by the framework, because the IndexedDB API only provides a handful filter possibilities. Also you can only use one filter when retrieving data.

Create a custom filter

To create a custom filter, the Linq2IndexedDB framework has an addFilter method. This method can be found in the linq namespace and accepts 3 arguments.

• The first argument is a name for the filter, this name will be used when you want to use your filter. Make sure that this name is unique over all the filters, if not an exception will be thrown.
• The second argument is a function that will be used to determine if the data is valid or not. When this function gets called 2 arguments will be passed. The first one is the object to validated. The second one is filter metadata. For example this object contains the name of the property you need to filter on, or an additional value provided, … The result of this function must be a Boolean value telling if the provided object is valid or not.
• The last argument is also a function and will be used to retrieve additional filter metadata. This function must return a new function with optionally arguments to retrieve the additional in formation. For example this information can be one or more values to use in the IsValid function. When called 3 arguments are passed to this function. The first one is a callback function that needs to be the return value of the function to retrieve the information. The second argument is the queryBuilder object that needs to be passed to the callback method. The third argument is the filterMetaData object. The additional information you retrieve, needs to be added to this object so you can use it later on in the IsValid method. After this is done, this object must also be provided as argument to the callback method.

In the code below I added an example of an equals filter.

   1: linq2indexedDB.prototype.linq.addFilter("equals", function (data, filter) {

   2:     return data[filter.propertyName] == filter.value;

   3: }

   4: , function (callback, queryBuilder, filterMetaData) {

   5:     return function (value) {

   6:         filterMetaData.value = value

   7:         return callback(queryBuilder, filterMetaData);

   8:     }

   9: });

When all this is done, you can start using this filter just by calling it after the where, or and and method call.

   1: db.from("objectStore").where("property").equals("value").select()

Anonymous filters

An other way to add a custom filter is by adding a callback function to the where, or or and method. When this callback function gets called, the object you need to validate is passed as an argument. The result of the callback function must be a Boolean value determining if the object is valid or not

   1: db.linq.from("objectStore").where(function (data) {

   2:     return data.Age > 3;

   3: }).and(function (data) {

   4:     return data.Age < 10;

   5: }).select();

conclusion

In the Linq2IndexedDB framework, you now have 2 ways to add custom filters. In the case you use the addFilter method, you can add a custom filter that can be reused and will appear in intellisense so you can easily call it. The other way provides an easy only once use of a filter. In both cases you are now flexible to provide your own filtering.

JSON: serialize and deserialize functions in JavaScript

In my Linq2IndexedDB project, I take advantage of the web workers to do all the filtering that the IndexedDB API doesn’t allow (multiple filters, like, inArray, …). For all these filters I have an “isValid” method which determines if the data satisfies the condition. So when I want to use these methods in my background worker, I need to make sure I can call them. For that I have 3 possibilities

1. Add a copy of all the filters in my background worker file
2. Include the file where my filters are defined
3. Serialize and deserialize the “isValid” functions

The first one wasn’t an option for me, because i wanted to keep all filter logic at one place. The second one was an option, but I didn’t want to use multiple JavaScript files for my library. And an other reason why I don’t like the first 2 is because developers would have to change my library if they want to add additional filters. So this left me only with the third possibility.

For this I’ve done some research. I already knew that you could call .toString on a function and this would result into a string representation of the function. And with the Function.call method I would be able to call the function in my background worker. But I didn’t wanted a solution that had to call the Function.call method in the background worker, I wanted to use the .isValid method that was provided in the filter objects. So I dug into the JSON API and found the following solution.

Serialize Objects

The JSON.stringify (the method that turns JavaScript objects into JSON text) accepts 2 arguments. The first argument is the object you want to turn into a JSON text and the second argument accepts a replacer function. This function gets called for every value in the object structure (even for properties in child objects) and accepts a key (name of the property) and a value (the value of the property) argument. The return value of the object is the object that will be stringified. By using this function, I can return the string representation of the function (in case of function) and return the value in the other cases.

   1: JSON.stringify(filters, function (key, value) {

   2:     if (typeof value === 'function') {

   3:         return value.toString();

   4:     }

   5:     return value;

   6: });

deserialize objects

The next thing we want to do, is deserialize the function again. Like the stringify method, the parse method (returns a JSON string into a JavaScript objects) also a callback (reviver) method as second argument. This method also gets called for every key and value for every level of the result. In this callback you can reform generic objects into instances of pseudo classes, strings into dates, strings into functions, …

   1: JSON.parse(filtersString, function (key, value) {

   2:     // reform your objects here

   3: });

Once I have the string representation of the function, I can start rebuilding that function. This is done by creating a new Function Object. The constructor of the Function objects accepts 2 arguments, a list of arguments for the function you want to create and the string representation of the body of the function you want to create.

   1: new Function(arguments, functionBody);

So if we put those 2 together we get the following:

   1: JSON.parse(filtersString, function (key, value) {

   2:     if (value 

   3:         && typeof value === "string" 

   4:         && value.substr(0,8) == "function") {

   5:         var startBody = value.indexOf('{') + 1;

   6:         var endBody = value.lastIndexOf('}');

   7:         var startArgs = value.indexOf('(') + 1;

   8:         var endArgs = value.indexOf(')');

   9:  

  10:         return new Function(value.substring(startArgs, endArgs)

  11:                           , value.substring(startBody, endBody));

  12:     }

  13:     return value;

  14: });

Detecting if the value is a function, is done by checking if the value is a string and starts with the word ‘function’. If this is the case, we will determine the arguments and the function body so we can pass it to the Function constructor. Retrieving the arguments of the new function is done by taking the string value between the first ‘(‘ and ‘)’. Retrieving the function body is done by getting the string value between the first ‘{‘ and the last ‘}’. Passing these 2 values to the constructor of the function will create a new function. This is the value that gets returned in case of a function. In all other cases we just return the value.

Conclusion

By using the stringify and parse method of the JSON API you can provide a generic way to serialize and deserialize your objects. Serializing functions can easily be done by calling the toString method on the function and deserializing a function can be done by using the Function constructor which is present in the JavaScript language.