Application Cache is not gone, oh my! Or, is it?

Reports of Application Cache’s early demise are false. Application Cache (a.k.a Cache Manifest and AppCache) isn’t perfect, it can be very frustrating to get working and many people hate it with passion, but the fact is that the API will be around for the foreseeable future. I’ve been getting many questions, so this blog post is an attempt to shed some light on a confusing topic.

Mozilla says Application Cache is deprecated and unstable, WTF?

First, let’s review what Mozilla is saying about Application Cache since the majority of questions I get come from what they are saying. Mozilla has some fairly strong wording on their developer site, you can read all about it here. Or, if you don’t feel like following a link, here’s the official text from Mozilla Developer Network (MDN) website. I added the underlines to emphasize wording that’s tripping people up:

Deprecated. This feature has been removed from the Web standards. Though some browsers may still support it, it is in the process of being dropped. Do not use it in old or new projects. Pages or Web apps using it may break at any time.

Using the application caching feature described here is at this point highly discouraged; it’s in the process of being removed from the Web platform. Use Service Workers instead. In fact, as of Firefox 44, when AppCache is used to provide offline support for a page a warning message is now displayed in the console advising developers to use Service workers instead (bug 1204581).

Mozilla also references the Web Hypertext Application Technology Working Group’s (WHATWG) position on Application Cache. But, their wording is similar but definitely different in meaning from the WHATWG statement in their HTML Living Standard, Section 7.7. You may want to read this several times:

This feature is in the process of being removed from the Web platform. (This is a long process that takes many years.) Using any of the offline Web application features at this time is highly discouraged. Use service workers instead.

An oversimplified description of the WHATWG is it’s an organization that provides research and makes recommendations to the W3C – World Wide Web Consortium. It’s the W3C that finalizes recommendations and makes them into standards.

Also note: Application Cache is, for the moment, still officially part of the latest draft of the WHATWG HTML Living Standard document as shown here in Section 7.2.2 (it’s a big document, it’s best to just search for application cache).

 

What is the true state of Service Workers?

Reality is always nuanced when it comes to cross-browser web application development. Here’s a summary of facts for cross-browser developers to consider (as of February 1, 2016):

  • Service Workers are NOT supported on the following platforms, and you’ll need to use Application Cache if it suits your requirements and any Application Cache bugs don’t adversely affect your apps. You can verify these on caniuse.com.
    • Android Browsers at v4.4.4 and older (there are still a lot of phones out there using older browsers). Yes, a percentage of these users download Chrome. However, as of Feb 1, 2016, Android is reporting that v4.1.x thru v4.4 represents 60.8% of all phones in circulation.
    • Desktop Safari
    • iOS Safari
    • IE and Edge – don’t forget that a lot of offline field worker apps are built specifically for laptops.
  • The Service Worker specification is still a Working Draft according to the W3C. It’s not final according to the official standards bodies. Mozilla even lists Service Workers as “Experimental Technology”. You can view that yourself here. Here’s MDNs wording copied directly from their webpage:

This is an experimental technology 

Because this technology’s specification has not stabilized, check the compatibility table for the proper prefixes to use in various browsers. Also note that the syntax and behavior of an experimental technology is subject to change in future versions of browsers as the spec changes.

  • Service workers are more complex and require coding skills in comparison to Application Cache which requires only a configuration file and one line of HTML code. Here’s an example. It’s going to be a big step for long time Application Cache developers to wrap their heads around using Service Workers. It’s a very powerful API but it’s not going to be as easy as simply swapping out your Application Cache for offline capabilities built on Service Workers.

What to do, what to do?

My thoughts are MDNs statement leaves developers between a rock and a hard place, and holds the most water if the ONLY web browsers you need to support are browsers that have implemented all aspects of Service Workers that meet your requirements. You’ll also need to keep in mind that Service Workers are in beta so they can also can break or not work as expected, and there is a remote possibility the functionality could change.

When you combine MDN’s statement with the WHATWG statement and the WHATWG Living Standard document the picture becomes more clear. Here’s my interpretation: Application Cache is going to be around for a while longer. However, it would be a really good idea for you, as a cross-browser developer, to keep your eye on the progress of Service Workers and to consider using them in applications if/when it makes sense based on requirements and browser capabilities.

Don’t throw away an working/stable code built on Application Cache just yet.   If you are building apps on platforms that don’t support Service Workers then Application Cache may be your only viable alternative until you can upgrade to newer platforms or other browsers.

Additional References:

Tags: , , , , ,
Posted in Browsers, JavaScript, Mobile, Offline | No Comments »

The W3C’s browser-based, JavaScript Geolocation API is excellent as a one-size-fits-all interface, but that approach comes at a price and it can cause some serious limitations when it comes to implementing more stringent professional, commercial and government use-cases.

Challenges with the default Geolocation API. One of the primary limitations is the Geolocation API does not tell you how it got a location. All locations are lumped together in a black box. Let me explain. On a smartphone or tablet, location data comes from one of three places: the GPS chipset, the cellular provider’s Location Service, or the browser’s Location Service. The W3C Geolocation API simply lumps these data points together. The end result is typically seen by the end user as significant and disturbingly wild jumps back and forth in the reported location, sometimes over large distances. A key to minimizing these fluctuations is to gain back control and understand which location provider created the latitude and longitude point.

Cordova-plugin-advanced-geolocation. The good news is that Android, in particular, has a very detailed API called LocationManager for examining geolocation data that comes from the device.  And, even better news for JavaScript developers is that the API, along with its access to all on-device GPS and Network location providers, has been exposed thru a Cordova plugin that is available here: https://github.com/Esri/cordova-plugin-advanced-geolocation.

What geolocation data is available? With this plugin you’ll be able to programmatically differentiate between the following geolocation data as well as get access to GPS satellites meta data:

  • Real-time GPS location – This is data from the on-board GPS or some devices will allow it to be the location data from an external GPS that is connected to the device via bluetooth.
  • Cached GPS location – Most devices cache the last-known GPS location and it’s persistent even when the device is restarted.
  • Real-time Network location triangulation – this is completely dependent on devices and cellular service providers. It may require WiFi to be turned on. It also may not be available in all countries or regions.
  • Cached Network location – Most devices cache the last-known network-based location and it’s persistent even when the device is restarted.

Use cases. With this plugin, you can now use your JavaScript skills to implement the following use cases and much more. For example, I’ve always wanted to play with the Satellite data to make a 3D map of the satellites using JavaScript. This plugin provides a huge advantage to developers building applications for capturing a single location such as field survey work as well as the following use cases and others that I haven’t thought of:

  • Determine a static outdoors location and only use GPS.
  • While indoors turn on only network location. Do not use GPS.
  • While in an urban area, use network location to get initial location before the GPS warms up and then turn off network location and only use GPS
  • Compare the differences between GPS and Network locations
  • …???

How does this plugin help minimize location fluctuations? This plugin comes with a configuration option for turning on a buffer. You can set the size of the buffer, each new geolocation from the device will be added to it, and then plugin will determine the geometric center based on all the locations in the buffer.

Are there any other advanced plugins? Yes, some Cordova plugins are focused on being activity based and will detect if you are walking, stopped, moving, etc. These plugins tend to work as apps that can be backgrounded. Feel free to browse the Cordova plug-in directory here.

Tags: , , , , , , ,
Posted in Android, GPS, JavaScript, Mobile, PhoneGap | 2 Comments »

Does using PhoneGap improve geolocation?

No, using PhoneGap or Cordova doesn’t change how you retrieve geolocation information from within a mobile application and it doesn’t improve accuracy. You still use the W3C HTML Geolocation API coding pattern.

The reason why is because PhoneGap runs the HTML/CSS/JS code in a native OS WebView. This is a chrome-less browser with mostly similar functionality as Chrome (Android) and Safari (iOS). There isn’t any additional functionality that would improve accuracy.

Some people have asked me if cordova-plugin-geolocation provides additional functionality. That plugin only works for devices that don’t already support geolocation and then it abides by the W3C Geolocation API spec.

The HTML Geolocation API in the browser also works offline. Yes, it’s true. You can use a web browser offline to get geolocation and even view a map without PhoneGap, here’s an example project that demonstrates the functionality in a number of ways. As long as the GPS on the device is enabled and working, and the user opt’s in at the geolocation-approval prompt then the application will receive GPS location data.

Are there any advantages to using PhoneGap for geolocation?

Yes, there are several potential advantages to using PhoneGap for geolocation. You can retrieve geolocation information while the device screen is locked and/or when the application is running in the background. You cannot do this with a typical web app. As soon as a browser is minimized a web app will stop running even if the GPS is still running.

Another advantage is you can write a PhoneGap plug-in to tap directly into the native OS location API. Taking this approach will give you a greater level of control over the information you are retrieving. And, you have coding tools and access to APIs to better manage the device’s battery life in a way that the HTML Geolocation API cannot deliver. In the case of Android you can tap into significantly greater information such as details on GPS satellites, granular information on device’s internet connection, as well as NMEA strings and more.

Tags: , , , , ,
Posted in GPS, HTML5, Mobile, PhoneGap | Comments Off on Does using PhoneGap improve geolocation?

Just because you wrapped an existing website in bootstrap doesn’t necessarily mean it’s ready for use on mobile devices. This is especially true if the website was originally designed for desktop browsers. Yes, bootstrap can significantly improve the user interface and make it flexible across multiple screen sizes. But it’s also up to you to roll up your sleeves and make sure the code behind the scenes is also worthy of being mobile-ready.

So, here are a three challenges to consider that will help keep your smartphone using visitors happy.

Challenge 1 – The internet connection on mobile devices is not as reliable as your home or office wired network. That’s a fact. Download speeds can and will vary significantly. The larger the website in MBs, the more links it has to download, the more non-optimized images then the longer it will take to render and become ready for use, especially on a mobile device. And mobile users are a very impatient bunch when it comes to sluggishness.

Size does matter with web sites. Smaller sized files download faster. The fewer number of files that make up a website also means faster downloads. Optimize, optimize and optimize some more. Minify files. Combine multiple files into one. Optimize images for web display.

Challenge 2 – Site navigation needs to be rethought and resized with mobile users in mind. Modern mobile devices use finger-based navigation, as opposed to high-precision mouse pointers. Teeny, tiny buttons or links that look cool on an ultra-high resolution MacBook retina screen positively suck when you are trying multiple times to click on them with your fingertip. On some websites using desktop navigational elements on your phone becomes like a macabre video game as you repeatedly play hit-or-miss with your fingertips.

Mobile websites should be finger lickin’ good. Okay, maybe you don’t really have to lick your fingers, but at least right size your navigational elements while keeping people’s fingers in mind. And fingers come in all shapes, sizes and levels of dexterity. Bootstrap can help with this.

Furthermore, your design and testing should work equally well in both portrait and landscape modes (phone right side up or phone on its side), and you should be able to switch back and forth seamless between the two modes in the same browser session.

Challenge 3 – Mobile devices are significantly more sensitive to browser memory leaks and bloated web pages. The mobile operating system will simply kill of any app that it deems to be using too much memory. And most of us are simply not good stewards at keeping our mobile browsers tuned up and happy. Browser caches grow huge and don’t get cleaned out regularly; we keep too many tabs open and probably have more information in our browser history than the library of congress. To further add to our woes, many of us let our phones run for weeks without a restart which can allow memory leakage to grow over time.

Tweak as many aspects of web page performance that your time will allow. Optimize old code by re-writing and striving to add new efficiencies. I know it may sound crazy, but if your site is particularly large and complex then consider creating focused, mobile-only sites that have scaled down content, rather than trying the one-size fits all approach. Smaller sites not only load faster but they are easier to navigate, take up less memory and typically perform better.

Tags: , , , ,
Posted in JavaScript, Mobile | Comments Off on You can’t just wrap old websites in bootstrap and call it a day

Easy automation of JavaScript form testing

If you are writing unit tests that provide coverage for HTML forms then there is an easy, pure JavaScript way to automate testing of the underlying code that works on modern browsers. The nice thing about this approach is you don’t have to manually load a file every time you run the tests. You still need to test the HTML interface components but that’s a topic for a different blog post.

The concept is straightforward in that you need to emulate the underlying functionality of an HTML form. The good news is you don’t have to programmatically create an HTML Form or Input element. Here’s the pattern you need to follow:

  1. Create an xhr request to retrieve the file. Be sure to set the response type as blob.
  2. Take the xhr.response and create a new File Object using the File API.
  3. Inject the File Object into a fake Form Object or,
  4. You can also use FormData() to create an actual Form Object.
  5. The fake Form Object is now ready to pass into your unit tests. Cool!

Here’s how you create a JavaScript FormData Object. Depending on what data your code expects you can add additional key/value pairs using append():

var formData = new FormData();
formData.append("files",/* file array */files);

And, here’s what the basic pattern looks like to retrieve the file, process it and then make it available for your unit tests:


var formNode; // Unit tests can access form node via this global

function retrieveFile(){

    var xhr = new XMLHttpRequest();
    xhr.open("GET","images/blue-pin.png",true); //set path to any file
    xhr.responseType = "blob"; 

    xhr.onload = function()
    {
        if( xhr.status === 200)
        {
            var files = []; // This is our files array

            // Manually create the guts of the File
            var blob = new Blob([this.response],{type: this.response.type});
            var bits = [blob,"test", new ArrayBuffer(blob.size)];

            // Put the pieces together to create the File.
            // Typically the raw response Object won't contain the file name
            // so you may have to manually add that as a property.
            var file = new File(bits,"blue-pin.png",{
                lastModified: new Date(0),
                type: this.response.type
            });

            files.push(file);

            // In this pattern we are faking a form object
            // and adding the files array to it.
            formNode = {
                elements:[
                    {type:"file",
                        files:files}
                ]
            };

            // Last, now run your unit tests
            runYourUnitTestsNow();
        }
        else
        {
            console.log("Retrieve file failed");
        }
    };
    xhr.onerror = function(e)
    {
        console.log("Retrieved file failed: " + JSON.stringify(e));
    };

    xhr.send(null);
}
Tags: , , , , , , ,
Posted in JavaScript, testing | Comments Off on Easy automation of JavaScript form testing

The Geolocation API is built into all modern mobile browsers and it lets you take either a quick, onetime snapshot, or you can get continuous location updates. Using the browser to get your approximate location is very, very cool, but it’s also fraught with many challenges. The vast majority of blog posts on this API talk about what it can do, this blog post focuses on how to best use it and understanding the data provided by the API.

To start things out, let’s take a quick look at a short list of some of the challenges when using the Geolocation API.

Challenge 1. You will not know where the location information is coming from. There’s no way to tell if it’s from the GPS, the cellular provider or the browser vendors location service. If you care about these things then the native Android SDK, for example, gives you a huge amount of control over what they call ‘location providers.’

Challenge 2. You cannot force the app to stay open. This means that the typical user has to keep tapping the app to keep it alive otherwise the screen will go to sleep and minimize your app.

Challenge 3. Speaking about minimizing apps, when the browser is minimized the geolocation capabilities stop working. If you have a requirement for the app to keep working in the background then you’ll need to go native.

Challenge 4. You’ll have very limited control over battery usage. Second only to the screen on your phone or tablet, the current generation of GPS chips are major energy hogs and can suck down your battery very quickly. Since the Geolocation API gives you very little control over how it works you cannot build much efficiency into your apps.

Challenge 5. Most smartphones and tablets use a consumer-grade GPS chip and antenna, and that limits the potential accuracy and precision. On average, the best possible accuracy is typically between 3 and 10 meters, or 10 – 33 feet. This is based on my own extensive experience building GPS-based mobile apps and working with many customers who are also using mobile apps. Under the most ideal scenario, the device will be kept stationary in one location until the desired accuracy number is achieved.

What’s it good for? Okay, you may be wondering what is browser-based geolocation good for? It’s perfect for very simple use cases that don’t require much accuracy. If you need to map manhole covers, or parking spaces, or any other physical things that are close together you’ll need a GPS device with professional-level capabilities.

Here are a few generic examples that I think are ideal for HTML5 Geolocation:

  • Simply getting your approximate location in latitude/longitude and converting it to a physical address.
  • Finding an approximate starting location for searching nearby places or things in a database or for getting one-time driving directions.
  • Determining which zip code, city or State you are in to enable specific features in the app.
  • Getting the approximate location of a decently sized geological feature such as a park, a building, a pond, a parking lot, a driveway, a group of trees, an intersection, etc.

What’s the best way to get a single location? The best way to get a single location is to not use getCurrentPosition() but to use watchPosition() and analyze the data for a minimum set of acceptable values.

Why? Because getCurrentPosition() simply forces the browser to barf up the best available raw, location snapshot right now. It literally forces a location out of the phone. Accuracy values can be wildly inaccurate, especially if the GPS hasn’t warmed up, or if you aren’t near a WiFi with your WiFi turned on, or if your cellular provider can’t get a good triangulation fix on your phone, or it returns a cached value from a different location altogether. There are many, many “what ifs?”

So, I recommend using watchPosition() and firing it off and letting it run until the return values meet some minimum criteria that you set. You need to know that while this is happening the location values returned may cover a fairly wide geographic area…remember our best accuracy values are 10 – 30 meters. Here’s a real-world example of Geolocation API location values that I captured over a 5 minute period while standing stationary in front of a building.

5 minute snapshot

What steps do you recommend? Here are five basic steps to help guide you towards one approach for getting the best location. This is a very simplistic approach and may not be appropriate for all use cases, but I think it’s adequate to demonstrate the basic concepts for working towards determining the best possible location.

Step 1. Immediately reject any values that have accuracy above a certain threshold that you determine. For example, let’s say we’ll reject any values with an accuracy reading greater than 50 meters.

Step 2. Create three arrays, one for accuracy, latitude and longitude. If the accuracy is below your threshold, or in this case < 50 meters, then push the values to the appropriate arrays. You will also need to set a maximize size for the array and create a simple algorithm for adding new values and removing old ones.

The array length could be 10, 20 or even 100 or more entries. Just keep in mind that the longer the array, the longer it will take to fill up and the longer the user will have to wait for the end result.

Step 3. Start calculating the average values for accuracy, latitude and longitude.

Step 4. Start calculating the standard deviation for accuracy, latitude and longitude.

Step 5. If your arrays fill up to the desired length and the average accuracy meets your best-possible criteria, and the standard deviation is acceptable then you can take the average latitude, longitude values as your approximate location.

For an example of this simple algorithm at work visit the following URL on your phone and step outside to get a clear view of the sky: http://esri.github.io/html5-geolocation-tool-js/field-location-template.html. [Updated link: Oct. 27, 2015]

Tags: , , , , , ,
Posted in Browsers, GPS | Comments Off on HTML5 Geolocation API: Getting the best single location