Offline JavaScript Part 3 – Intermittent Offline

This is Part 3 of my offline JavaScript series and it covers intermittently offline web apps. The vast majority of web apps are built on the false assumption that the internet will always be available. Yes, the internet is available the vast majority of the time, and most of us rarely encounter issues. However when, not if but when, the internet fails most web apps simply crash and burn in fairly spectacular fashion. I suggest a different approach that there are many, many common use cases that can benefit from offline capabilities in both consumer and professional apps.

As discussed in Part 1, intermittently offline web apps are designed to gracefully handle the occasional, temporary internet connection hiccup. The goals of an intermittent offline app are to make the offline capabilities are lightweight, invisible to the user, and allow the user to seamless pass thru a temporary loss of data connectivity.

The good news, as discussed in Part 2, is you can use a variety of libraries and APIs to solve many of the challenges related to partial offline including detecting whether or not you have an internet connection, and handling of http requests while offline.

How do I decide if I need intermittent offline capabilities?

If you answer ‘yes’ to the following question then you need to consider adding offline capabilities:

Does the app have any critical functionality that could fail if the internet temporarily goes down?

Critical functionality means functionality that’s important to your core business. And to be realistic I’m not talking about building fully armored applications that take every possible contingency into account. That’s just not feasible for the vast majority of non-military-grade applications. Some of the most common use cases are filling in forms and requesting data. And, temporary interruptions can be vary anywhere from a few seconds to a few minutes or longer, and they can happen once or multiple times.

If your application can’t handle this and it needs to then making changes to allow it to be offline can make a big difference to the user. It’s almost as if web development should have it’s own version of “Do no harm” or something like “we can do our best to make users lives easier.” You might be surprised that some very simple and common use cases can benefit from being offline enabled such as filling in form data, or reading an online article.

Filling in form data. This has probably happened to everyone who uses the internet and it applies to both retail/consumer and commercial applications. You spend a while filling out a detailed web form only to have the submit fail and destroy all your hard work because of a temporary interruption in the internet connection or something simply went wrong between the app and the web server.

If our form data was offline-enabled we could store the form data in LocalStorage before attempting to send the data to the server. We could also temporarily prevent the web form from submitting and notify the user there is no internet connection.

Reading an online article. In this scenario you are reading an article while waiting for a train.  Once you get on the train you know the internet will be marginal. You accidentally click on navigation link while scrolling down and the new page fails to load. This effectively ruins your browsing experience because the new page failed to load and you can’t go back to the previous page because it wasn’t cached..

There are a number of different ways to protect this type of application. The easiest way is to block any page load requests until the internet is restored. You can also take advantage of the built-in browser cache to store HTML, CSS, images and JavaScript.

Show me an example workflow?

The most basic workflow takes into consideration the following questions. How these questions get answered depends on your requirements.

  • Do you allow users to restart apps while offline?
  • Do you simply block all HTTP requests and lock down the app?
  • Do you queue HTTP requests and their data?
  • Do you pre-cache certain data?
  • How will you detect if the app is online or offline?

Here is an example coding pattern for the most basic intermittent offline workflow:

What about Offline/Online detection?

If you have no control over what browsers your customers choose, then my recommendation is to use a pre-built library such as Offline.js to check if the internet connection is up or down. It’s not perfect but it’s the best choice out there as of the writing of this post.

Don’t only rely on the window.navigator.online property. It has too many inconsistencies and it is only marginally reliable if the general public is using your app.

What about caching?

There are several built-in browser caching mechanisms that can help your app get past the occasionally internet hiccup. When your app goes offline, you’ll have to rely on local, in-browser resources to keep things going:

  • Browser Caching
  • LocalStorage
  • IndexedDB

As mentioned above, browser caching can be a very efficient way to store HTML, JavaScript, images and CSS. Depending on how you set up your web server, this caching takes place automatically in the users browser and can represent a huge performance gain in eliminating HTTP round trips. I’m not going to talk much about this because there are a ton of great online resources already out there.

Using LocalStorage involves writing JavaScript code if you want to temporarily store HTTP requests. It’s limited to String-based data, so if you are using Objects or binary data you’ll have to serialize the data when you write it to LocalStorage and deserialize when you read it out. LocalStorage also almost always has a limit in terms of how much storage is available. 5MB is the commonly accepted limit.

IndexedDB, on the other hand, stores a wide variety of data types and can store significantly more than 5MB. While in theory the amount of storage space available to IndexedDB is unlimited, practical application of it on a mobile device limits you to around 50MB – 100MB. Your mileage may vary depending on available device memory, the current memory footprint of the browser and the phone’s operating system.

IndexedDB can work natively with types String, Object, Array, Blob, ArrayBuffer, Uint8Array and File. This offers a huge pre- and post-processing savings if you simply are able to pass data directly into IndexedDB.

There are also a number of abstraction libraries that wrap LocalStorage and IndexedDB such as Mozilla’s localForage. These types of libraries are great if you have requirements to store 5MBs of data or less. If your app is running a browser that doesn’t support IndexedDB or WebSQL (e.g. Safari), and you need more than 5MBs of space then you’ll have problems. One potential advantage of some of these libraries is that some of them provide their own internal algorithms for serializing and deserializing data. If working directly with algorithms isn’t your thing, then a library like this can be a huge benefit.

Can you show me some code?

Yes! Here is a very simple example of how to implement basic offline detection into your apps. It’s easiest to try it in Firefox since you can quickly toggle it online/offline using the File > Work Offline option.

The code is available at: https://jsfiddle.net/agup/1yxj5mzp/. You’ll notice two things when you go offline. First is that jsfiddle, itself, will detect you are offline in addition to the web app code. When you go to click the Get Data button while offline, the code sample should detect you are offline and fire off a JavaScript alert.

<!DOCTYPE html>
<html>
<head lang="en">
    <meta charset="UTF-8">
    <title>Simple Offline Demo</title>
</head>
<body>
<div id="status">Status is:</div>
<button onclick="getData()">Get Data</button>
<!-- This is our Offline detection library -->
<script src="https://github.hubspot.com/offline/offline.min.js"></script>

<script>

    // Set our options for the Offline detection library
    Offline.options = {
        checkOnLoad: true,
        checks: {
            image: {
                url: function() {
                    return 'https://esri.github.io/offline-editor-js/tiny-image.png?_='
                        + (Math.floor(Math.random() * 1000000000));
                }
            },
            active: 'image'
        }
    }

    Offline.on('up', internetUp);
    Offline.on('down',internetDown);

    var statusDiv = document.getElementById("status");
    statusDiv.innerHTML = "Status is: " + Offline.state;

    function getData() {

        // See if internet is up or down
        Offline.check();

        switch (Offline.state) {
            case "up":
                // If the internet is up go ahead and retrieve data.
                getFeed(function(success,response){
                    if(success){
                        alert(response);
                    }
                })
                break;
            case "down":
                alert("DOWN");
                break;
        }
    }

    function getFeed(callback) {
        var req = new XMLHttpRequest();
        req.open("GET",
                "https://tmservices1.esri.com/arcgis/rest/services/LiveFeeds/Earthquakes/MapServer?f=pjson");
        req.onload = function() {
            if (req.status === 200 && req.responseText !== "") {
                callback(true,req.responseText);
            } else {
                console.log("Attempt to retrieve feed failed.");
                callback(false,null);
            }
        };

        req.send(null);
    }

    function internetUp(){
        console.log("Internet is up.");
        statusDiv.innerHTML = "Status is: up";
    }

    function internetDown(){
        console.log("Internet is down.");
        statusDiv.innerHTML = "Status is: down";
    }
</script>
</body>
</html>

Are there any examples of real-life offline apps or libraries?

The github repository offline-editor-js is a full-fledged set of libraries for taking maps and mapping data offline and it’s being used in commercial mapping applications around the world. It includes a variety of sample applications that demonstrate how applications can work in either intermittently or fully offline mode.

Wrap-up

Hopefully you have seen that common use cases can significantly benefit from having basic offline capabilities. Modern browsers have advanced to the point where it’s fairly easy to build web apps that can survive intermittent interruptions in the internet. Taking advantage of these capabilities can offer a huge benefit to your end users.

Resources

Optimizing content efficiency – HTTP caching
Offline-editor-js – Offline mapping library

Going Offline with HTML5 and JavaScript, Part 1

There are two primary use cases for going offline with mobile HTML5 web applications and JavaScript: partial offline and full offline. Before diving into building offline apps, understanding the differences between these use cases will help you build the best applications for your requirements. The functionality in modern browsers has finally gotten to the point where it is feasible (and fun!) to build offline web applications.

Partial Offline. Partial offline means the vast majority of the time the application is online, however it can continue to work if the internet connection gets temporarily interrupted. A partially offline app understands that requests for remote resources, that is resources that don’t exist on the device, will automatically defer to local resource, or at least fail gracefully, during the period of time when an internet connection doesn’t exist. Partially offline apps typically cannot be reloaded or restarted offline. The coding required to handle this scenario is much lighter weight than the architecture required for going fully offline. An example of partial offline is a reader app that pre-caches certain HTML pages of your choice. If the internet connection gets disrupted you can continue reading and navigating between the cached pages.

The partially offline scenario exists because there is no such thing as a perfect internet connection for mobile. In reality, internet connections and download speeds are very choppy when measured over a period of minutes or hours. Sure, some internet providers market 4G connections as being extremely fast, or have the best coverage etc., etc., blah. The bottom line is cellular and even WiFi internet connections are not guaranteed. A good example of this is coffee shops. They don’t come with an uptime guarantee, so if a couple of yahoos sitting next to you are streaming HD Netflix then that will surely bring the internet connection to its knees.

In reality, if you don’t live danger close to a cell phone tower and are moving around doing your job or running errands chances are your internet connection will fluctuate up and down over time. Anyone that owns a smartphone has experienced this at one time or another. Dropped calls are perfect example. You may be shopping and going in and out of buildings, or hanging out in the back of a taxi, sitting in your car pulled off the side of the road, or perhaps even just watching your kids as they play in the neighborhood park. A web application architected for partially offline will let you keep surfing or working for a short period of time, and hopefully long enough until the internet connection comes back up.

Full Offline. A fully offline JavaScript application is one that starts out online to download all the necessary data and files, then it can be completely disconnected from the internet. These apps can survive browser restarts and reloads, they can stay offline indefinitely and/or they can be resynced online at some point in the future.

Fully offline apps need to be architected in a much more robust way than their partially offline cousins. Partial offline apps can be considered more fragile than fully offline apps because they can’t be restarted or reloaded while offline, and you have to be very careful to limit their capabilities while offline otherwise the user can easily break the app. Fully offline apps are built with the knowledge that they may be offline for extended periods of time and need to be self-sufficient because users will be depending on them. If a fully offline app breaks then the user will be completely hosed (and very unhappy) until some point in the future where the internet connection can be restored and they can resync the app.

Offline apps can break in any number of interesting ways such as throwing a 404 when the user hits the back button or simply crashing when the app unsuccessfully attempts to a load a new page. By their very nature, fully offline apps may have larger and more complex data storage and life cycle requirements. They cache entire HTML web pages and all their associated JavaScript libraries, images and supporting data.

Examples of full offline apps include mapping apps, web email, reader apps, and any other apps that require information from a database.  User data is typically downloaded locally, stored on disk and then accessed by offline web applications. Any data that’s stored in memory will be lost when if the device or browser is restarted while offline.

 

Web offline versus Native offline

When building out your requirements, it’s a best practice to do an honest comparison between offline web capabilities and the offline capabilities of native SDKs.  In general as of the writing of this post, it’s fair to say that native apps still offer much more robust offline capabilities than the latest versions of mobile browsers. There are a few exceptions where browsers may have similar capabilities but almost always the level of control is more limited.

Native apps have the advantage because they basically have direct access to the device operating system and many of the capabilities are simply integrated into the respective SDKs. Here is a partial list of capabilities that are commonly seen in native offline requirements:

Web apps, on the other hand, run within the browser and are subject to any limitations imposed by the browser. The browser, itself, is a native app and it restricts it’s own children (web apps) to certain security restrictions. A few examples of web app limitations include:

  • Access to a limited number of censors. Access is not consistent across different browser types.
  • Limited control over location services via HTML5 Geolocation API.
  • JavaScript cannot programmatically read and write non-cached files on the device without user intervention.
  • Internet connectivity detection typically dependent on 3rd party libraries such as offline.js. Support is inconsistent across some platform/browser combinations.
  • Indirect and limited control over battery life and optimization.
  • Browsers and any of their associated tabs stop running as soon as the browser is minimized. If you have a requirement for the app to “wake up” from a minimized state under certain conditions you will have to go native.

 

Summary

Partial offline applications are design to continue working gracefully during intermittent interruptions in connectivity. Because offline is considered a temporary or even momentary condition in this use case, partial offline apps can use lighter weight architecture and have smaller data storage needs than full offline apps.

Fully offline apps are designed to be taken offline for extensive periods of time. They have to meet more demanding requirements and need a comprehensive architecture that enables storing of HTML files, JavaScript libraries, and user data as well as being able to handle browser reloads and restarts while offline.

Lastly, when having conversations about building offline apps you should weight web versus native offline capabilities against your requirements. Native SDKs still offer much richer control over most of the aspects of offline functionality.

Stay tuned for additional posts on this subject. Part 2 will look at the features and APIs you can use to take applications offline.

 

References

10 ways to deal with intermittent internet connections

How accurate is Android GPS? Part 1: Understanding Location Data

Wikipedia – Browser Security

How Accurate is HTML5 Geolocation, really? Part 2: Mobile Web

Where Part 1 focused on non-GPS enabled devices, Part 2 is totally focused on mobile web geolocation. The great news is that the usage of HTML5 location services along-side the fact that there is a GPS chipset in most, if not all, modern smartphones and tablets dramatically improves the chances of getting an accurate location. And, besides that fact — mobile geolocation is simply a lot of fun to work with.

I also want to point out that there are an increasing number of really good blog posts covering the topic of “how to use” the API that look at the nitty-gritty of how the code works. This post is different in that I’ve tried to focus on “how to build successful applications” with the API, and how to get the most out of the API so that you can successfully implement your unique requirements.

What’s different about desktop vs. mobile HTML5 Geolocation? With mobile you can access the GPS if it’s available. It’s important to note that in order to access a device GPS you have to set the optional enableHighAccuracy property in your code. Contrary to what is shown in some samples on the internet, you can use this property with both the getCurrentPosition() and watchPosition() functions.

//One time snapshot
navigator.geolocation.getCurrentPosition(
     processGeolocation,
     // Optional settings below
     geolocationError,
     {
         timeout: 0,
         enableHighAccuracy: true,
         maximumAge: Infinity
     }
);

//Tracking users position
watchId = navigator.geolocation.watchPosition(
     processGeolocation,
     // Optional settings below
     geolocationError,
     {
         timeout: 0,
         enableHighAccuracy: true,
         maximumAge: Infinity
     }
);

How accurate is it??? This is the million dollar question, right? When using enableHighAccuracy() on a phone where all the appropriate permissions have been selected and granted, I’ve typically seen accuracy readings as low as 3 meters (~10 feet) that were obtained within 10 – 30 seconds of kicking off the geolocation functionality. I’d consider that excellent for most consumer and retail applications. You should be aware that like any location-based functionality you will get spurious (abnormal) results that fall way outside the norm, and sometimes these results are wildly wrong.

I’ve seen claims that using the enableHighAccuracy() property slows down the phones ability to deliver a location. I’m going to argue that those claims are misleading. It is true that the GPS, itself, can take a significant amount of time to warm up and start delivering high accuracy results. For an in-depth look at that topic see my post on the Six Most Common Use Cases for Android GPS. However, there are conditions where simply enabling the enableHighAccuracy() property doesn’t affect the speed in which you can get the initial result. More on these topics below.

What is the best way to try out various configuration scenarios? I’ve built an HTML5 Geolocation Testing tool that can be used in browser, or it can be repurposed to work in PhoneGap or Titanium. It is a jQuery-based mobile application that includes a map and settings view were you can adjust all the different properties and try out different configuration scenarios. It’s a work-in-progress so I welcome suggestions and pull requests.

 Why HTML5 Geolocation rather than native? Applications using HTML5 Geolocation typically have slightly different requirements than native GPS-based applications. Each platform has its advantages and disadvantages and it all comes down to your requirements, budget, timeframes and skill sets:

  • Ability to re-use existing JavaScript and HTML5 skills to build a high-accuracy mobile application.
  • Don’t have access to native platform developers or skillsets on Android, iPhone and/or Windows Phone.
  • Need a cross-platform stand-alone web app, or a web app that has been repurposed to work with PhoneGap or Titanium.
  • Quickly locate the user/consumer within a reasonable expectation of accuracy.
  • Typically it is a non-commercial, consumer grade application that does not have extremely high accuracy requirements (e.g. < 1 meter).

How fast can I get an initial location result? The answer is very fast, potentially within a few seconds, given the following scenarios:

  • If there was a cached GPS or Network location stored on the phone. The GPS location is, of course, from the GPS chipset. The Network location comes from your wireless carrier and is dependent on your phone and their capabilities.
  • How the timeout and maximumAge properties are set. If you set timeout = 0 and maximumAge = Infinity it will force the application to grab any cached location, if one is available. Other settings may result in delays.
  • If the phone or tablet has decent internet connectivity and Wifi enabled.
  • If the device is in an urban area with many wifi nodes broadcasting their SSIDs nearby.
  • The device has a clear and uninterrupted view of the sky. GPS’s listen for a very weak signal from multiple satellites. These signals can be partially or completely blocked by buildings, thick foliage, vehicle roofs, etc.

 How accurate is the initial location result? Hah, you might have guessed I’d say that it depends.  When you first kick off a geolocation request, accuracy does depend on a number of different factors that are mentioned above. And it’s safe to say that, in the vast majority of cases, the first location is not the most accurate and typically not the most dependable. If you want the fastest, most accurate location possible then you will most likely need to either do multiple snapshots, or use watchLocation until your desired level of accuracy is met. It’s important to note because I’ve been asked about this many times, you cannot expect the GPS, itself, to have enough time to lock onto a satellite and deliver a fast, accurate initial location. It may take dozens of seconds or even minutes. Yep, it’s true. Factors that affect initial location accuracy include:

  • Cached locations – how recently the user accessed location functionality. For example, applications like Facebook typically grab a location when you open the app. So frequent users of social media are more likely to have a fresh, cached location that non-social media users. If you are targeting business travelers, the cached location might the last city before they got on a plane. Or, it could be your home neighborhood and not where you work or go to games.
  • Wifi turned “on”. If the Wifi is turned on then the device can access the location service and there is a much greater chance that the initial result is fairly accurate. If you didn’t have a chance to read Part 1, when the Wifi is on your browser gathers local Wifi node information from your Wifi card, and it can use that information in a location service provider request over the internet to try and triangulate your position. Typically this means your initial location can be within a block or two of the actual position. Also, it is possible if Wifi is turned on that you can get a significantly more accurate initial location than if you were using GPS by itself with no Wifi or internet.
  • Internet connectivity strength. If you have a poor internet connection and no Wifi, then the browser’s requests to the location service can be delayed, blocked or even interrupted.
  • No VPN. Take note commercial application developers: as mentioned in Part 1, if VPN software is in use it can wildly affect accuracy and even place you in another State (or Country).

Can I use HTML5 Geolocation for mobile tracking? Yes, with caveats. Typically HTML5 tracking applications are built inside a native wrapper framework such as PhoneGap or Titanium. There are several immediate problems with stand-alone, browser-only HTML5 tracking applications. First, there is no built-in functionality to keep the screen from going to sleep. Second, when the screen goes to sleep the HTML5 Geolocation functionality also goes to sleep. Native-based tracking applications can work around these limitations and listen passively in the background when they are minimized. Third, you have little control over the GPS settings to help management battery consumption.

Can I use HTML5 Geolocation offline? Yes! If there is no cellular connection or Wifi available, then HTML5 Geolocation can still access cached locations and real-time GPS information. This is vastly different from what was discussed in Part 1 as related to applications targeted at laptops, desktops and tablets that may or may not have GPS. If a device does not have a built-in or externally available GPS then your offline application will not work.

Handling abnormal location results. Your application will occasionally encounter widely inaccurate results and you need to handle these gracefully for the best user experience possible. My recommendation is to check the timestamps and distance traveled between the current geolocation object and the previous one. If the distance or speed seems excessive then you’ll need to reject the result. In the reference section below is a link to more information on calculating the distance between two points containing latitude and longitude. As an example, see the attached screenshot with the spurious results indicated by red circles. Also note in the screenshot the accuracy level was 3 meters, so it’s important to understand that even at high accuracy levels you still need to very that each location meets your minimum requirements. This way your results will always look polished and professional to the end user.

Spurious results

What are some of the downsides of using HTML5 Geolocation versus native? The bottom line is that for simple location gathering and basic tracking HTML5 Geolocation is just fine. This should meet the requirements for most consumer applications. For anything more complex than that you should consider looking at going native.

  • It may not work on older phones and older browsers (depending on your definition of old). See below in the references section for a link to a fallback library to handle these situations.
  • HMTL5 Geolocation offers significantly less control over GPS settings. This can have an unacceptable impact on more complex applications.  Because of this, I also suggest that HTML5 Geolocation is not suitable for long-running tracking applications.
  • Battery life management. This is a direct result of bullet #2. It’s more challenging to manage battery life with HTML5 Geolocatoin if your requirements call for continuous use of the GPS.  Your control is very limited with respect to these two properties: timeout and maximumAge.
  • Cannot use it when the application is minimized. If your requirements calls for the ability to passively receive locations while in a minimized state then, as mentioned earlier, you will have to go native.
  • Very little control over how often you want location updates. You’ll need to do a bunch of custom coding to emulate what is already built into native application APIs. For example, the native Android API offers very detailed control over what type of geolocation data you can get access to, how you can access it and how often. Read more on that topic in my post on How Accurate is Android GPS Part 1 – Understanding Location Data and also take a look at Android’s LocationManager Class.

References

W3C Geolocation API Specification 

HTML5 Geolocation Test Tool

Mozilla – Using Geolocation

Calculating distance between two points.

Geolocation fallback library for older browsers