Indexed DB: Defining your structure, the new way

Some months ago I posted a blog post about defining your database structure. Because Indexed DB is still in draft, specifications can change from time to time. This is the case for the way you want to define your database structure. As I am typing now the only browser that is currently implementing this new specification is FireFox Nightly.
The way you create object stores and indexes hasn’t changed, but the way you can change it did. Changing your database structure still happens in a VERSION_CHANGE transaction, but the developer won’t be able to manually create a new VERSION_CHANGE transaction. Instead, the version of database you want to use needs to be provided when opening a database connection. When the database doesn’t have the version you requested, an onupgradeneeded event will occur. In this event you will be able to handle all your database changes like creating and deleting objectstores, creating and deleting indexes, …
But this all means that the SetVersion method for creating a VERSION_CHANGE transaction will become obsolete. So this means you will always have to define your database structure before you can start using them, even for indexes.
So how does this new way look like?

   1: var req = window.indexedDB.open(databaseName, databaseVersion);

   2: req.onsuccess = function (e) {

   3:     // Opening of the databas in the given version was succesfull

   4: }

   5:  

   6: req.onerror = function (e) {

   7:     // Error while opening the database

   8: }

   9:  

  10: req.onupgradeneeded = function (e){

  11:     // Event for handeling the database structure changes

  12: }

  13:  

  14: req.onblocked = function (e){

  15:     // Handles the database open request while the database gets upgraded

  16: }

The onblocked event was also added with the onupgradeneeded event. The onblock event will handle attempts to open the database while the database is upgrading to a more recent version. This way you can notify the user the request was blocked and they need to retry later.

As last there is an onversionchange event added on the databse object. This event will be called when the database is about to upgraded to a newer version. This way you’ll be able to close your current connection so you don’t get an error later on.

   1: var req = window.indexedDB.open(databaseName, databaseVersion);

   2:  

   3: req.onsuccess = function (e) {

   4:     req.result.onversionchange = function(){

   5:         // Close your connection

   6:     }  

   7: }

Offline Application Caching: Make your web application offline available

In my previous post I have been talking about IndexedDB, an in-browser database. This means, once we can persist data on the client, it’s possible to make our web application offline available. For this we have the offline application caching API.

When we want to take our application offline, the first thing we need to do is creating a manifest file. The main function of this file is describing which files have to be offline available. These files will be downloaded when the users visits the web application for the first time. Also when the manifest file has been changed since the last visit, these files will be downloaded again.

Manifest file

The beginning of the manifest file is always the same:

CACHE MANIFEST

Below this it’s recommended to place a comment with some kind of version number/date. This is necessary when you want to force the browser to download the files again because you changed one or more files. This is because browsers will only update their application cache when the manifest file is changed and not if one of the resources described in the file change. This means whenever you change a file described in the manifest file, you should update the version number in a comment.

# v3 2012-01-11

Now we have three parts in the manifest file. The first one is the explicit section. This means all files that are described in this section will be downloaded. This is also the default section, so it isn’t necessary to provide the ‘CACHE:’ title explicitly. Note that the colon after the title of the section is required. In the explicit section all kinds of files are allowed, for example: html files, images, js files, … Every resource you want to describe, takes one line.

CACHE:
/Detault.htm
/Scripts/jquery-1.7.1.min.js
/Css/ui-lightness/jquery-ui-1.8.16.custom.css
/Css/ui-lightness/images/ui-bg_diagonals-thick_18_b81900_40x40.png

The next part of the manifest file will describe which files should never be cached. This is the case for pages who relay on code that is executed on the server. For example a logon page. In this section you can use the “*” character as wildcard.That’s a fancy way of saying that anything that isn’t in the appcache can still be downloaded from the original web address, as long as you have an internet connection. This also means all resources on the webpage will be cached, even if they are hosted on an other domain.

The wildcard is important if we want to provide an “open-ended” offline web application. This is common for large web applications, whom we want to make offline available such as Wikipedia.

NETWORK:
/Logon.apsx
/Secure
*

The last part of the file will describe the fallback mechanisms. If you are working offline, and you request a resources that isn’t offline available, the configured resources will be shown.

FALLBACK:
/ /HTML/Offline.htm

Offline available webpages

All webpages we want to use offline need the manifest attribute with a reference to the manifest file. This way the browser knows where he can locate the file describing the offline resources.

<html manifest="/cache.manifest">

It doesn’t matter where the manifest file is located on the server. The only thing you need is the correct path to it. There are 2 extensions that are commonly used for the manifest file. These are “.manifest” and “.appcache”. It doesn’t matter which one you use, as long as the extensions are recognized by your webserver. In IIS 7.0 or higher the a MIME-type for .manifest exists.

If u are using an Apache webserver, you can use the AddType directive in your side-wide httpd.conf by adding the following lines:

AddType text/cache-manifest .appcache "access plus 0 seconds"
AddType text/cache-manifest .manifest "access plus 0 seconds"

To avoid the risk of caching the manifest files, it is a good idea to set expires headers on your web server for manifest files so they expire immediately. In Apache you can configure this as follows:

ExpiresByType text/cache-manifest "access plus 0 seconds"

For IIS 6.0 you can find the explanation here.

Application cache events

The offline application cache API is also provided with some events:

Event	Description
Checking	Fires when the browsers notices a manifest attribute in the html tag, even in case you already visited the page
Downloading	Fires if the browser starts downloading the files described in the manifest file
Progress	Fires periodically while downloading.Contains information about the number of files that have been downloaded and the number of files that are still queued
Cached	Fires when all the files in the manifest file are downloaded. The web app is now fully cached and ready to use offline. This is only the case if it’s the first time data from the manifest file is downloaded.
Noupdate	Fires if you visit an offline-enabled page and the manifest file hasn’t changed.
Updateready	Fires when downloading of the files described in the manifest file was successful. The new version web app is now fully cache. This is the case if the data from the manifest file has ever been downloaded in the past.
Error	Fires when ever something goes wrong. Possible causes: - HTTP error 404 (Page not found) - HTTP error 410 (Permanently Gone) - Page failed to download properly - Manifest file changed while updating - Browser failed to download one of the resources listed in the manifest file
Obsolete	The manifest was not found. Possible causes: - HTTP error 404 (Page not found) - HTTP error 410 (Permanently Gone) This means the application cache has been deleted.

It’s a best practice to call the window.applicationCache.SwapCache() method when the Updateready event is fired. This forces the browser to switch to the most recent application cache. If you forget to do this, the user needs to reload the webpage in order to take advantage of the new version.

Browser State

In the window object we have an interface “NavigatorOnline” which provides us information about the state of the browser. If the attribute “onLine” is false, we can be sure the browser is definitely offline. In case the attribute is true, the browser might be online, but that isn’t for sure.

window.navigator.onLine

There are also 2 events. The online and offline event that are fired when the browser either goes online or offline. You can attach the events on the following ways:

• using the addEventListener on the window, document or document.body
• Setting the .ononline or .onoffline properties on document or document.body to a JS function. (For some reason the window.ononline and window.onoffline will not work)
• By specifying the ononline or onoffline attributes on the <body> element in the HTML.

Note when using the first 2 solutions, you can only attach the events after the page load event.

Indexed DB: Reading multiple records

In my previous post I have been talking about reading data. Today, I’ll be talking about reading multiple records at once. Here for we will use a cursor. A cursor is a transient mechanism used to iterate over multiple records in a database. The storage operations of the cursor can be used on the underlying index or an object store.

In the IDBObjectStore interface, we have the openCursor method to create a new cursor for retrieving data. In the IDBIndex interface, we have 2 ways to create a new cursor. These methods are openCursor to retrieve the values from the index and openKeyCursor to retrieve the keys. There are 2 optional parameters that can be provided when calling these methods. The first parameter is an IDBKeyRange, with this we will narrow the result by defining the bounds of the keys we want to retrieve. The second parameter is the direction the cursor must navigate trough the results.

IDBKeyrange

A key range is a continuous interval over some data type used for keys. A key range can have one of the following situations:

lower bounded: The keys must have a value smaller than the provided lower bound

upper bounded: The keys must have a value larger than the provided upper bound

lower and upper bounded: The keys must have a value between the lower and the upper bound

unbounded: All keys will be valid

Single value: The key must be the provide value

The upper and lower bound can be open, this means the value of the bound won’t be included, or closed, this means the value of the bound will be included.

More information about the IDBKeyRange interface can be found here. You will also find some more information about the methods to create a key rage. If you want to use an unbounded key range, you don’t need to provide a key range.

Retrieving data with a cursor

As for all actions preformed on the database a transaction is also needed in case of reading data with a cursor.

   1: var txn = dbconnection.transaction([“ObjectStoreName”]);

   2: var objectStore = txn.objectStore(“ObjectStoreName”);

   3: var cursorReq;

   4: // IE 10, Chrome and Firefox implementation

   5: if(window.msIndexedDB || window.mozIndexedDB || window.webkitIndexedDB){

   6:     cursorReq = store.openCursor();

   7: }

   8: // IE Indexed DB Prototype implementation

   9: else{

  10:     cursorReq = store.openCursor(IDBKeyRange.lowerBound(0));

  11: }

  12:  

  13: handleCursor(cursorReq, txn, success, error);

First things first, we define the cursor. One of the first thing you notice is that the IE Indexed DB prototype requires a key range. This means it won’t be possible to use the unbounded key range. Because I’m using an auto increment key for my object store, I can take use of the lowerbound method to create a key range from 0 to forever. After the cursor is defined, I use a method to handle the reading of the cursor.

   1: function handleCursor(cursorReq, txn, success, error){

   2:     cursorReq.onsuccess = function (event) {

   3:         if (event.result) {

   4:             var cur = event.result;

   5:             if (cur) {

   6:                 cursor_get_record(cur);

   7:             }

   8:         }

   9:         else if (cursorReq.result) {

  10:             var cur = cursorReq.result;

  11:             // Present the data

  12:             cur.continue();

  13:         }

  14:     cursorReq.onerror = error

  15: }

Again the IE Indexed DB prototype has a different implementation for handling a cursor. I’ll start with explaining the correct way. If the request object contains a result. If this is empty, you reached the end of the cursor. If the result is an object, the value of the current record in the cursor can be found in the value field. When you handled the value, you can navigate to the next record in the cursor by calling the continue function on the result object.

When working with the Indexed DB prototype, you have to make another approach. That’s why I created a recursive function for it. (cursor_get_record). The only parameter I need to pass is the result object I get out of the parameter that is provided when the call for the cursor was successful.

   1: function cursor_get_record(cur){  

   2:     cur.move(); 

   3:     if(cur.value)

   4:     {

   5:         // Present the data

   6:         cursor_get_record(cur);

   7:     }

   8: }

With the move method we fetch the next record from the cursor. If the cursor contains a value, you can handle it and afterwards you call the current method to fetch the next record of cursor.

Indexed DB: Reading data

Finally we get to the part where we can see the results of our Indexed DB tutorial. For the last few moths we have been structuring our database, adding data and now I’ll show how you can read data from an Indexed DB. For this we have 2 approaches. First one is retrieving a single record and the other way is retrieving a collection of records by a cursor.

In this post I’ll handle the retrieving of a single record. A post about retrieving data with a cursor will appear in the near future.

Retrieving data from an object store

In the IObjectStore interface we have an get method to retrieve data. This method must be provided with a key we want to search on. The key parameter can be a valid key or a key range. In case you don’t provide a valid key, a DATA_ERR exception will be thrown.

In the first situation a record of the object store will be retrieved where the given key is the same as the key of the key/value pair. In the other situation, the first record will be retrieved that matches one of the keys defined in the key range. If no record gets found, the result will be undefined. If you are using the Indexed DB prototype on IE 9, you’ll get an “Object with specified key not found.” error in stead of an result.

Time for some code:

When we want to read data, we need to create a transaction. This we do on the same way as for adding data. The only difference is we can use the IDBTransaction.READ_ONLY mode. This is the type that will be used as default when we create a transaction.

   1: var txn = dbconnection.transaction([“ObjectStoreName”]);

   2: var objectStore = txn.objectStore(“ObjectStoreName”);

The result of the get method is a IDBrequest object. When this is successful, the record is added to the result.

   1: var getReq = store.get(key);

   2: getReq.onsuccess = function(event) { 

   3:     var result;

   4:     // IE 9 implementation

   5:     if(event.result){

   6:         result = event.result;

   7:     }

   8:     // IE 10, Chrome and Firefox implementation

   9:     else if (getReq.result){

  10:         result = getReq.result;

  11:     }

  12:     // Code to present the result on your screen

  13: };

Retrieving data from an Index

In a previous post I have been talking about Indexes. Indexes make it possible to look up data from the object store by fields of the value object. By example: we have an object store person. In this object store we store persons. A person object has a first name and a last name field. By creating an index with a key path “last name”, we make it possible to retrieve data from the object store by providing the last name of the person.

In the IDBIndex interface we have the get method to retrieve a record from the object store, and the GetKey method to retrieve the key of the record from the object store. In both cases a key must be provided. This key parameter can be a valid key or a key range. For the rest every thing works the same as for retrieving data from an object store.

One think you need to keep in mind. In the Indexed DB prototype on IE9 the get and getKey method work different. In this case the getKey method retrieves the data and the get method the key of the record.

   1: var txn = dbconnection.transaction([“ObjectStoreName”]);

   2: var objectStore = txn.objectStore(“ObjectStoreName”);

   3: var index = store.index(“IndexName”);

   4: var getReq = index.getKey(key);

   5:  

   6: getReq.onsuccess = function(event) { 

   7:     var result;

   8:     if(event.result){

   9:         result = event.result;

  10:     }

  11:     else{

  12:         result = getReq.result

  13:     }

  14:     // Code to present the result on your screen

  15: };

Indexed DB: Manipulating data

The first thing we need to do when we want to read data, is to make sure we have data available in our database. So this post will handle inserting and updating data into the database.

For inserting data we have 2 possibilities: we have an add and a put method available. Both methods take the same parameters, but there is a slight difference between them. When we use the add method, we need to be sure that no other object with the same key is already added in the database. If it does, an constraint error (CONSTRAINT_ERR) will occur. So the add method will only be used when we want to add data that won’t be overwritten when it is already present with the same key in the database.

The put method we will usually use when we want to update data. If the data isn’t present yet with the same key, the data will be inserted. If you want some more information about the steps for storing a record into an object store, you can find this here on the W3C specs site.

Make sure when you are starting a new transaction to manipulate data, you will use a read_write transaction. Otherwise you will receive an READ_ONLY_ERR when calling the methods.

The put and the add method are both available in the IObjectStore interface. This means once we have created the transaction where we will work in, we need to retrieve the object store we want to work on. We do this by calling the ObjectStore method on the transaction. The only thing we need to pass is the name of the object store.

var txn = dbconnection.transaction([“ObjectStoreName”]
                                                 , IDBTransaction.READ_WRITE);
var objectStore = txn.objectStore(“ObjectStoreName”);

Once we have the object store object, we can start adding data to it. The result of the add operation is an IDBRequest object. This means the onsuccess will be called when the operation was successful and an onerror will be called when something went wrong.
When the onsuccess function gets called, the result of this action will be the key of the object in the object store

var addresult = objectStore.add(data);
addresult.onsuccess = function (event) {
      // Adding data is successful
      // Commit the transaction when using IE9
      var key;

      // IE 9 implementation
      if(event.result){
            key = event.result
      }
      // IE 10, Chrome and Firefox implementation
      else if (addresult.result){
            key = addresult.result;
      }
};
addresult.onerror = function (event) {
      // Handle the error
}

If we use a put for adding or changing data, we get the same structure only we will use the put method instead.

var putresult = objectStore.put(data);
putresult.onsuccess = function (event) {
      // Putting data is successful
      // Commit the transaction when using IE9
};
putresult.onerror = function (event) {
      // Handle the error
}

Now that we have added some data to our database, we can start retrieving it and present it to our users…

Indexed DB: Deleting your database

In a previous post I have been talking about creating and deleting a database. The creation/opening of a database is supported in all the major browsers, but deleting isn’t. Only Internet Explorer currently supports deleting your database trough the IDBFactory interface.

Because we have been playing for a while now, it’s interesting we can delete our database so we can rebuild it from scratch without having to use a new name for our database. I’m only used to work on windows machines, so the solution I will propose will probably only work under windows. If you are using an other operating system, it will work on the same way, but you’ll have to look where you can find the databases.

Firefox

The Indexed DB databases of Firefox can be found on the following location:

<location of the windows user profiles>\<account name>\AppData\Roaming\
Mozilla\Firefox\Profiles\<some random characters>.default\IndexedDB

In my case this is:

C:\Users\kristof\AppData\Roaming\Mozilla\Firefox\Profiles\ tvv6t475.default\indexedDB

You will find a folder with your current domain. Delete these folder, and you can start all over again. In my case, I get the following:

http+++localhost+50350

I have to notice that the AppData folder is a hidden folder, so it is possible you won’t see it when going to your user profile directory.

Chrome

The Indexed DB databases of Chrome can be found on the following location:

<location of the windows user profiles>\<account name>\AppData\Local\Google\
Chrome\User Data\Default\IndexedDB

In my case this is:

C:\Users\kristof\AppData\Local\Google\Chrome\User Data\Default\IndexedDB

You will find a folder and a file with your current domain. Delete these files, and you can start all over again.

In my case I get the following:

folder: http_localhost_50350.indexeddb.leveldb
file:     http_localhost_50350.indexeddb

Indexed DB: Transactions

Today, I’ll handle the transaction subject. As said in previous posts, every request made to the database needs to be done in a transaction. So for every read or write request we need to create a new transaction. There for we need a database connection and 2 argument that we will pass to the transaction method.

The first argument will define the scope of the transaction. Here we pass all the object stores we want to use during the transaction. We do this by passing the object store names in an array. Providing an empty array will allow to use all available object stores in the database.

The second argument is the mode we want to use to access the data. This is an optional parameter and if not provided the transaction will be created read only by default. If you want to manipulate data, you’ll need to pass IDBTransaction.READ_WRITE. There is also a third mode, CHANGE_VERSION, but this type of transaction can only be created in the setVersion method. More about this method can be found in my post about Indexed DB: Defining the database structure

var txn = dbconnection.transaction([], IDBTransaction.READ_WRITE);

txn.oncomplete = function () {
     // Transaction successful
};
txn.onabort = function () {
    // Code to handle the abort of the transaction
};
txn.onerror = function () {
// Code to handle the error
};

For a transaction we have 3 possible outcomes. The first one is that the transaction is committed and so got completed. This function will be called if the transaction was successful and in the case we were reading data, we can here write the code to show the retrieved data it in the browser. Keep in mind that a transaction in the Indexed DB API is committed by default, but in IE9 you still need to do this manually.

The onabort function will be called when we manually call the abort function on the transaction object. This means the transaction must do a rollback.

The last one will handle all the errors that can occur within the transaction, this will also mean that the transaction must rollback.

Now that we know how to create a transaction, we can start retrieving and manipulating data from our database. This will be the next subject of my future posts.