Application Cache is not gone, oh my! Or, is it?

Reports of Application Cache’s early demise are false. Application Cache (a.k.a Cache Manifest and AppCache) isn’t perfect, it can be very frustrating to get working and many people hate it with passion, but the fact is that the API will be around for the foreseeable future. I’ve been getting many questions, so this blog post is an attempt to shed some light on a confusing topic.

Mozilla says Application Cache is deprecated and unstable, WTF?

First, let’s review what Mozilla is saying about Application Cache since the majority of questions I get come from what they are saying. Mozilla has some fairly strong wording on their developer site, you can read all about it here. Or, if you don’t feel like following a link, here’s the official text from Mozilla Developer Network (MDN) website. I added the underlines to emphasize wording that’s tripping people up:

Deprecated. This feature has been removed from the Web standards. Though some browsers may still support it, it is in the process of being dropped. Do not use it in old or new projects. Pages or Web apps using it may break at any time.

Using the application caching feature described here is at this point highly discouraged; it’s in the process of being removed from the Web platform. Use Service Workers instead. In fact, as of Firefox 44, when AppCache is used to provide offline support for a page a warning message is now displayed in the console advising developers to use Service workers instead (bug 1204581).

Mozilla also references the Web Hypertext Application Technology Working Group’s (WHATWG) position on Application Cache. But, their wording is similar but definitely different in meaning from the WHATWG statement in their HTML Living Standard, Section 7.7. You may want to read this several times:

This feature is in the process of being removed from the Web platform. (This is a long process that takes many years.) Using any of the offline Web application features at this time is highly discouraged. Use service workers instead.

An oversimplified description of the WHATWG is it’s an organization that provides research and makes recommendations to the W3C – World Wide Web Consortium. It’s the W3C that finalizes recommendations and makes them into standards.

Also note: Application Cache is, for the moment, still officially part of the latest draft of the WHATWG HTML Living Standard document as shown here in Section 7.2.2 (it’s a big document, it’s best to just search for application cache).

 

What is the true state of Service Workers?

Reality is always nuanced when it comes to cross-browser web application development. Here’s a summary of facts for cross-browser developers to consider (as of February 1, 2016):

  • Service Workers are NOT supported on the following platforms, and you’ll need to use Application Cache if it suits your requirements and any Application Cache bugs don’t adversely affect your apps. You can verify these on caniuse.com.
    • Android Browsers at v4.4.4 and older (there are still a lot of phones out there using older browsers). Yes, a percentage of these users download Chrome. However, as of Feb 1, 2016, Android is reporting that v4.1.x thru v4.4 represents 60.8% of all phones in circulation.
    • Desktop Safari
    • iOS Safari
    • IE and Edge – don’t forget that a lot of offline field worker apps are built specifically for laptops.
  • The Service Worker specification is still a Working Draft according to the W3C. It’s not final according to the official standards bodies. Mozilla even lists Service Workers as “Experimental Technology”. You can view that yourself here. Here’s MDNs wording copied directly from their webpage:

This is an experimental technology 

Because this technology’s specification has not stabilized, check the compatibility table for the proper prefixes to use in various browsers. Also note that the syntax and behavior of an experimental technology is subject to change in future versions of browsers as the spec changes.

  • Service workers are more complex and require coding skills in comparison to Application Cache which requires only a configuration file and one line of HTML code. Here’s an example. It’s going to be a big step for long time Application Cache developers to wrap their heads around using Service Workers. It’s a very powerful API but it’s not going to be as easy as simply swapping out your Application Cache for offline capabilities built on Service Workers.

What to do, what to do?

My thoughts are MDNs statement leaves developers between a rock and a hard place, and holds the most water if the ONLY web browsers you need to support are browsers that have implemented all aspects of Service Workers that meet your requirements. You’ll also need to keep in mind that Service Workers are in beta so they can also can break or not work as expected, and there is a remote possibility the functionality could change.

When you combine MDN’s statement with the WHATWG statement and the WHATWG Living Standard document the picture becomes more clear. Here’s my interpretation: Application Cache is going to be around for a while longer. However, it would be a really good idea for you, as a cross-browser developer, to keep your eye on the progress of Service Workers and to consider using them in applications if/when it makes sense based on requirements and browser capabilities.

Don’t throw away an working/stable code built on Application Cache just yet.   If you are building apps on platforms that don’t support Service Workers then Application Cache may be your only viable alternative until you can upgrade to newer platforms or other browsers.

Additional References:

Offline JavaScript Part 3 – Intermittent Offline

This is Part 3 of my offline JavaScript series and it covers intermittently offline web apps. The vast majority of web apps are built on the false assumption that the internet will always be available. Yes, the internet is available the vast majority of the time, and most of us rarely encounter issues. However when, not if but when, the internet fails most web apps simply crash and burn in fairly spectacular fashion. I suggest a different approach that there are many, many common use cases that can benefit from offline capabilities in both consumer and professional apps.

As discussed in Part 1, intermittently offline web apps are designed to gracefully handle the occasional, temporary internet connection hiccup. The goals of an intermittent offline app are to make the offline capabilities are lightweight, invisible to the user, and allow the user to seamless pass thru a temporary loss of data connectivity.

The good news, as discussed in Part 2, is you can use a variety of libraries and APIs to solve many of the challenges related to partial offline including detecting whether or not you have an internet connection, and handling of http requests while offline.

How do I decide if I need intermittent offline capabilities?

If you answer ‘yes’ to the following question then you need to consider adding offline capabilities:

Does the app have any critical functionality that could fail if the internet temporarily goes down?

Critical functionality means functionality that’s important to your core business. And to be realistic I’m not talking about building fully armored applications that take every possible contingency into account. That’s just not feasible for the vast majority of non-military-grade applications. Some of the most common use cases are filling in forms and requesting data. And, temporary interruptions can be vary anywhere from a few seconds to a few minutes or longer, and they can happen once or multiple times.

If your application can’t handle this and it needs to then making changes to allow it to be offline can make a big difference to the user. It’s almost as if web development should have it’s own version of “Do no harm” or something like “we can do our best to make users lives easier.” You might be surprised that some very simple and common use cases can benefit from being offline enabled such as filling in form data, or reading an online article.

Filling in form data. This has probably happened to everyone who uses the internet and it applies to both retail/consumer and commercial applications. You spend a while filling out a detailed web form only to have the submit fail and destroy all your hard work because of a temporary interruption in the internet connection or something simply went wrong between the app and the web server.

If our form data was offline-enabled we could store the form data in LocalStorage before attempting to send the data to the server. We could also temporarily prevent the web form from submitting and notify the user there is no internet connection.

Reading an online article. In this scenario you are reading an article while waiting for a train.  Once you get on the train you know the internet will be marginal. You accidentally click on navigation link while scrolling down and the new page fails to load. This effectively ruins your browsing experience because the new page failed to load and you can’t go back to the previous page because it wasn’t cached..

There are a number of different ways to protect this type of application. The easiest way is to block any page load requests until the internet is restored. You can also take advantage of the built-in browser cache to store HTML, CSS, images and JavaScript.

Show me an example workflow?

The most basic workflow takes into consideration the following questions. How these questions get answered depends on your requirements.

  • Do you allow users to restart apps while offline?
  • Do you simply block all HTTP requests and lock down the app?
  • Do you queue HTTP requests and their data?
  • Do you pre-cache certain data?
  • How will you detect if the app is online or offline?

Here is an example coding pattern for the most basic intermittent offline workflow:

What about Offline/Online detection?

If you have no control over what browsers your customers choose, then my recommendation is to use a pre-built library such as Offline.js to check if the internet connection is up or down. It’s not perfect but it’s the best choice out there as of the writing of this post.

Don’t only rely on the window.navigator.online property. It has too many inconsistencies and it is only marginally reliable if the general public is using your app.

What about caching?

There are several built-in browser caching mechanisms that can help your app get past the occasionally internet hiccup. When your app goes offline, you’ll have to rely on local, in-browser resources to keep things going:

  • Browser Caching
  • LocalStorage
  • IndexedDB

As mentioned above, browser caching can be a very efficient way to store HTML, JavaScript, images and CSS. Depending on how you set up your web server, this caching takes place automatically in the users browser and can represent a huge performance gain in eliminating HTTP round trips. I’m not going to talk much about this because there are a ton of great online resources already out there.

Using LocalStorage involves writing JavaScript code if you want to temporarily store HTTP requests. It’s limited to String-based data, so if you are using Objects or binary data you’ll have to serialize the data when you write it to LocalStorage and deserialize when you read it out. LocalStorage also almost always has a limit in terms of how much storage is available. 5MB is the commonly accepted limit.

IndexedDB, on the other hand, stores a wide variety of data types and can store significantly more than 5MB. While in theory the amount of storage space available to IndexedDB is unlimited, practical application of it on a mobile device limits you to around 50MB – 100MB. Your mileage may vary depending on available device memory, the current memory footprint of the browser and the phone’s operating system.

IndexedDB can work natively with types String, Object, Array, Blob, ArrayBuffer, Uint8Array and File. This offers a huge pre- and post-processing savings if you simply are able to pass data directly into IndexedDB.

There are also a number of abstraction libraries that wrap LocalStorage and IndexedDB such as Mozilla’s localForage. These types of libraries are great if you have requirements to store 5MBs of data or less. If your app is running a browser that doesn’t support IndexedDB or WebSQL (e.g. Safari), and you need more than 5MBs of space then you’ll have problems. One potential advantage of some of these libraries is that some of them provide their own internal algorithms for serializing and deserializing data. If working directly with algorithms isn’t your thing, then a library like this can be a huge benefit.

Can you show me some code?

Yes! Here is a very simple example of how to implement basic offline detection into your apps. It’s easiest to try it in Firefox since you can quickly toggle it online/offline using the File > Work Offline option.

The code is available at: http://jsfiddle.net/agup/1yxj5mzp/. You’ll notice two things when you go offline. First is that jsfiddle, itself, will detect you are offline in addition to the web app code. When you go to click the Get Data button while offline, the code sample should detect you are offline and fire off a JavaScript alert.

<!DOCTYPE html>
<html>
<head lang="en">
    <meta charset="UTF-8">
    <title>Simple Offline Demo</title>
</head>
<body>
<div id="status">Status is:</div>
<button onclick="getData()">Get Data</button>
<!-- This is our Offline detection library -->
<script src="http://github.hubspot.com/offline/offline.min.js"></script>

<script>

    // Set our options for the Offline detection library
    Offline.options = {
        checkOnLoad: true,
        checks: {
            image: {
                url: function() {
                    return 'http://esri.github.io/offline-editor-js/tiny-image.png?_='
                        + (Math.floor(Math.random() * 1000000000));
                }
            },
            active: 'image'
        }
    }

    Offline.on('up', internetUp);
    Offline.on('down',internetDown);

    var statusDiv = document.getElementById("status");
    statusDiv.innerHTML = "Status is: " + Offline.state;

    function getData() {

        // See if internet is up or down
        Offline.check();

        switch (Offline.state) {
            case "up":
                // If the internet is up go ahead and retrieve data.
                getFeed(function(success,response){
                    if(success){
                        alert(response);
                    }
                })
                break;
            case "down":
                alert("DOWN");
                break;
        }
    }

    function getFeed(callback) {
        var req = new XMLHttpRequest();
        req.open("GET",
                "http://tmservices1.esri.com/arcgis/rest/services/LiveFeeds/Earthquakes/MapServer?f=pjson");
        req.onload = function() {
            if (req.status === 200 && req.responseText !== "") {
                callback(true,req.responseText);
            } else {
                console.log("Attempt to retrieve feed failed.");
                callback(false,null);
            }
        };

        req.send(null);
    }

    function internetUp(){
        console.log("Internet is up.");
        statusDiv.innerHTML = "Status is: up";
    }

    function internetDown(){
        console.log("Internet is down.");
        statusDiv.innerHTML = "Status is: down";
    }
</script>
</body>
</html>

Are there any examples of real-life offline apps or libraries?

The github repository offline-editor-js is a full-fledged set of libraries for taking maps and mapping data offline and it’s being used in commercial mapping applications around the world. It includes a variety of sample applications that demonstrate how applications can work in either intermittently or fully offline mode.

Wrap-up

Hopefully you have seen that common use cases can significantly benefit from having basic offline capabilities. Modern browsers have advanced to the point where it’s fairly easy to build web apps that can survive intermittent interruptions in the internet. Taking advantage of these capabilities can offer a huge benefit to your end users.

Resources

Optimizing content efficiency – HTTP caching
Offline-editor-js – Offline mapping library

Offline JavaScript Part 2 – Overview of Interfaces and APIs

In Part 1 we looked at the differences between partial and fully offline use cases. Part 2 provides an overview of the HTML5 Interfaces and JavaScript APIs that make it possible to go offline with web applications. Going offline involves working with multiple pieces and coding for specific patterns. I’ve tried my best to stick to technology that is widely available across the largest variety of browsers.

Offline dependencies

Offline web applications are dependent on three things.  It doesn’t matter if your application is partially or fully offline, you’ll still need to address these in your code.

  • Caching HTML, CSS and JavaScript
  • Data Storage
  • Offline/Online detection

Caching

Application Cache. The Application Cache, or AppCache, interface lets you specify and store HTML and CSS files as well as JavaScript libraries so that they are available from the browser’s native cache. Once an item is in the cache the browser will use it regardless of whether it’s online or offline. It’s almost like you never went offline!

The AppCache is an essential part of your application strategy for allowing offline browser reloads or restarts. Without this an application will simply fail to re-load while offline.

Data Storage

Browsers have a variety built-in JavaScript APIs for storing data. The data can be for maintaining the applications state such as for storing bookmarks and form data. Or, it can be used for storing information such as maps, address and phone lists, TO-DOs or points of interest for a vacation.

LocalStorage. The LocalStorage API is super-easy to use. It stores Strings in simple key/value pairs. It’s limited to about 5MBs on most browsers. The two main challenges you’ll run into with LocalStorage are hitting the storage limit and performance hits when serializing and deserializing data.

IndexedDB. IndexedDB is essentially an asynchronous noSQL database that lets you store a wide variety of datatypes so that you don’t have to deal with serialization/deserialization.  Datatypes include String, Object, Array, Blob, ArrayBuffer, Uint8Array and File. While many online sources will tell you that there isn’t a size limit, I’ll tell you that in general you should limit your storage on a mobile device around 50 – 100MBs to help prevent the browser from crashing.

WebSql. It’s widely recommended that you not build applications directly on WebSql. The World Wide Web Consortium (W3C) is letting this standard die off in favor of IndexedDB and LocalStorage. I’m really only including this here for reasons such as Safari 7 and and the Android native browsers before 4.4 only support WebSql. For more information on how to get around this read down to the section on IndexedDBShim.

3rd Party Browser Storage

If the built-in browser storage capabilities aren’t meeting your needs you still have other options.

IndexedDBShim. IndexedDBshim is a Polyfill for WebSQL-based browsers. Because IndexedDB isn’t natively supported on older versions Safari 7 and Opera you can use this 3rd party shim to transparently translate your IndexedDB code to work across Android and iOS.

PouchDB. PouchDB is an Open Source experimental library that is an attempt to smooth some of IndexedDB’s rough edges as well as provide additional functionality, such as the ability to sync with remote stores.

LocalForage (Mozilla).  LocalForage is also an attempt to bridge the gap between LocalStorage and IndexedDB. It gives you an interface that provides much wider browser coverage than IndexedDB by itself.  One of the downsides is the amount of storage you can use. If a user is on an older browser such as IE8 that’s limited to LocalStorage then that user will be limited to storing about 5MBs of data. If you requirements call for using more than that, such as downloading large address lists, then the app won’t work on that browser or you’ll have to build in some sort of paging mechanism that deletes the old data and brings in the new.

Offline/Online Detection

There are a number of ways to detect if the browser is online or offline as well as when the internet status changes.

NavigatorOnline.online.  Some browsers have a built-in detection mechanism. However, it is not always reliable and false positives are a distinct possibility. For that reason, you will have to build additional detection capabilities or lean towards a 3rd party library.

Offline.js. Offline.js is a small Open Source library (~3KB) that detects when you lose an internet connection and when it comes back up. While not perfect, it does handle a lot of cross-browser compatibility issues for you. And, if you find bugs you can always create a fix and submit pull requests.

References

Caniseuse – IndexedDB

Caniuse – LocalStorage

Caniuse – WebSQL

Let’s Take This Offline

Going Offline with HTML5 and JavaScript, Part 1

There are two primary use cases for going offline with mobile HTML5 web applications and JavaScript: partial offline and full offline. Before diving into building offline apps, understanding the differences between these use cases will help you build the best applications for your requirements. The functionality in modern browsers has finally gotten to the point where it is feasible (and fun!) to build offline web applications.

Partial Offline. Partial offline means the vast majority of the time the application is online, however it can continue to work if the internet connection gets temporarily interrupted. A partially offline app understands that requests for remote resources, that is resources that don’t exist on the device, will automatically defer to local resource, or at least fail gracefully, during the period of time when an internet connection doesn’t exist. Partially offline apps typically cannot be reloaded or restarted offline. The coding required to handle this scenario is much lighter weight than the architecture required for going fully offline. An example of partial offline is a reader app that pre-caches certain HTML pages of your choice. If the internet connection gets disrupted you can continue reading and navigating between the cached pages.

The partially offline scenario exists because there is no such thing as a perfect internet connection for mobile. In reality, internet connections and download speeds are very choppy when measured over a period of minutes or hours. Sure, some internet providers market 4G connections as being extremely fast, or have the best coverage etc., etc., blah. The bottom line is cellular and even WiFi internet connections are not guaranteed. A good example of this is coffee shops. They don’t come with an uptime guarantee, so if a couple of yahoos sitting next to you are streaming HD Netflix then that will surely bring the internet connection to its knees.

In reality, if you don’t live danger close to a cell phone tower and are moving around doing your job or running errands chances are your internet connection will fluctuate up and down over time. Anyone that owns a smartphone has experienced this at one time or another. Dropped calls are perfect example. You may be shopping and going in and out of buildings, or hanging out in the back of a taxi, sitting in your car pulled off the side of the road, or perhaps even just watching your kids as they play in the neighborhood park. A web application architected for partially offline will let you keep surfing or working for a short period of time, and hopefully long enough until the internet connection comes back up.

Full Offline. A fully offline JavaScript application is one that starts out online to download all the necessary data and files, then it can be completely disconnected from the internet. These apps can survive browser restarts and reloads, they can stay offline indefinitely and/or they can be resynced online at some point in the future.

Fully offline apps need to be architected in a much more robust way than their partially offline cousins. Partial offline apps can be considered more fragile than fully offline apps because they can’t be restarted or reloaded while offline, and you have to be very careful to limit their capabilities while offline otherwise the user can easily break the app. Fully offline apps are built with the knowledge that they may be offline for extended periods of time and need to be self-sufficient because users will be depending on them. If a fully offline app breaks then the user will be completely hosed (and very unhappy) until some point in the future where the internet connection can be restored and they can resync the app.

Offline apps can break in any number of interesting ways such as throwing a 404 when the user hits the back button or simply crashing when the app unsuccessfully attempts to a load a new page. By their very nature, fully offline apps may have larger and more complex data storage and life cycle requirements. They cache entire HTML web pages and all their associated JavaScript libraries, images and supporting data.

Examples of full offline apps include mapping apps, web email, reader apps, and any other apps that require information from a database.  User data is typically downloaded locally, stored on disk and then accessed by offline web applications. Any data that’s stored in memory will be lost when if the device or browser is restarted while offline.

 

Web offline versus Native offline

When building out your requirements, it’s a best practice to do an honest comparison between offline web capabilities and the offline capabilities of native SDKs.  In general as of the writing of this post, it’s fair to say that native apps still offer much more robust offline capabilities than the latest versions of mobile browsers. There are a few exceptions where browsers may have similar capabilities but almost always the level of control is more limited.

Native apps have the advantage because they basically have direct access to the device operating system and many of the capabilities are simply integrated into the respective SDKs. Here is a partial list of capabilities that are commonly seen in native offline requirements:

Web apps, on the other hand, run within the browser and are subject to any limitations imposed by the browser. The browser, itself, is a native app and it restricts it’s own children (web apps) to certain security restrictions. A few examples of web app limitations include:

  • Access to a limited number of censors. Access is not consistent across different browser types.
  • Limited control over location services via HTML5 Geolocation API.
  • JavaScript cannot programmatically read and write non-cached files on the device without user intervention.
  • Internet connectivity detection typically dependent on 3rd party libraries such as offline.js. Support is inconsistent across some platform/browser combinations.
  • Indirect and limited control over battery life and optimization.
  • Browsers and any of their associated tabs stop running as soon as the browser is minimized. If you have a requirement for the app to “wake up” from a minimized state under certain conditions you will have to go native.

 

Summary

Partial offline applications are design to continue working gracefully during intermittent interruptions in connectivity. Because offline is considered a temporary or even momentary condition in this use case, partial offline apps can use lighter weight architecture and have smaller data storage needs than full offline apps.

Fully offline apps are designed to be taken offline for extensive periods of time. They have to meet more demanding requirements and need a comprehensive architecture that enables storing of HTML files, JavaScript libraries, and user data as well as being able to handle browser reloads and restarts while offline.

Lastly, when having conversations about building offline apps you should weight web versus native offline capabilities against your requirements. Native SDKs still offer much richer control over most of the aspects of offline functionality.

Stay tuned for additional posts on this subject. Part 2 will look at the features and APIs you can use to take applications offline.

 

References

10 ways to deal with intermittent internet connections

How accurate is Android GPS? Part 1: Understanding Location Data

Wikipedia – Browser Security

Deleting an HTML Application Cache

When you are testing web applications that use an Application Cache, also sometimes called the manifest file, you have to delete this file every time you make a change to the application. If you don’t then none of the changes you make to the application will show up. The very purpose of the Application Cache is to semi-permanently store your HTML, CSS, JavaScript and images. It’s becoming increasingly popular for speeding up web app performance, and a requirement for taking web apps offline. In fact, Google now uses an application cache for their home page.

Simply trying to delete your browser cache in the normal way won’t necessarily clear the Application Cache and its associated files. So here’s a quick rundown that will hopefully save you some time.

Chrome – browse to chrome://appcache-internals/.  There may be a number of different caches listed. Select ‘Remove’ for any cache that you want to go bye-bye.

Chrome (Mobile Android) – go to Settings > Privacy (under Advanced) > CLEAR BROWSING DATA, checkbox the ‘Clear the cache’ option and then select the ‘Clear’ button.

IE 10 – go to Tools > Internet Options > Settings > Caches and databases tab. Select the cache that you want to delete and the click the ‘Delete’ button.

Safari (Mobile) – For Safari iPhone and iPad go to Settings and select “Clear Cookies and Data.”

Safari (Desktop) – Simply attempting Develop > Empty Caches may not work. On a Mac you may have to: close your browser, manually delete the .db file by going to //library/Caches/com.Apple.Safari and move any item ending in .db to the trash, then restart browser. If this doesn’t work then try restarting your machine. Yep, it’s an awful workflow and it’s been a known bug in Safari dating back to at least version 6.

Firefox (Desktop) – go to Tools > Options > Advanced > Network > Offline data > Clear Now.

Want to learn more about Application Cache’s? Here’s a good technical overview from WHATWG describing what is an application cache. And, MDN has a good article on Using the application cache.