Making coding changes and reloading locally hosted web pages over and over again is a pattern familiar web developers world wide. Another familiar pattern is to constantly wonder if your changes are caching in the browser and not being properly reflected in what you are seeing.

Fear not…there is a very easy fix for this and it doesn’t involve using the browsers empty cache options every single time between page reloads. Simply tell your local web server to send the browser a “no-cache” pragma directive in the HTTP header and then you should be good-to-go.

Once you make this change every web page you serve locally will automatically refresh, every single time. Here’s what the W3C has to say about no-cache headers:

 When the no-cache directive is present in a request message, an application SHOULD forward the request toward the origin server even if it has a cached copy of what is being requested. 

Make the change in Apache. Here’s how you make the change in your /etc/apache2/httpd.conf file on the latest Mac OS running 10.8+. Depending on how your machine is set up you can run the command “sudo pico httpd.conf” then enter your admin password and use the short cuts listed at the button of the pico window or use the ‘up’ and ‘down’ buttons on your keyboard to navigate around the file. Typically, the following text is pasted below any other ‘filesMatch’ tags that may reside in the configuration file. Once you are done be sure to restart apache. On Mavericks the command is “sudo apachectl start”:

<filesMatch "\.(html|htm|js|css)$">
    FileETag None
<ifModule mod_headers.c>
    Header unset ETag
    Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate"
    Header set Pragma "no-cache"
    Header set Expires "Wed, 11 Jan 1984 05:00:00 GMT"
</ifModule>
</filesMatch>

Make the change on IIS 7. If you want to make the change on Windows 7, Windows 2008/2008R2 or Vista then here is a link to Microsoft Technet. If you are using IIS Manager, in Step 4 choose the expire immediately options. Or, if you are using the command line copy this line and run it:

    <b>appcmd set config /section:staticContent /clientCache.cacheControlMode:DisableCache</b>

If you have some other operating system version hopefully you get the idea from the suggestions above and apply similar changes for your system.

Optimizing your own public web server cache settings. One last note, the no-cache header setting is typically only used in a development environment. To get the best page performance for your visitors you should allow some browser caching. If you want to learn more about optimizing browser caching here is a good article.

References

Optimizing Headers (Google)
RFC 2616, Section 14 Header Field Definitions
Configure HTTP Expires Response Header (IIS 7)
Manipulating HTTP headers with htaccess (you can make the same no-cache header change in httpd.conf in Mavericks)

Tags: , , , ,
Posted in Browsers | No Comments »

Debunking the myth of the one second web page load rule

Ever since Google announced that every mobile web site had to achieve less than one second loading times, I’ve been meaning to do a fun, psuedo-scientific study to help start the discussion of putting this well intentioned goal into a perspective that everyday developers and businesses can understand. This topic has been heavily promoted, with most of the blog posts simply trumpeting agreement and either implying or coming right and saying that any page that loads over one second is pretty much worthless. In case you’ve never read the actual article, here’s a qoute from Google:

“we must deliver and render the above the fold (ATF) content in under one second, which allows the user to begin interacting with the page as soon as possible.”

I propose that most consumers will put up with a majority of sites that don’t deliver in exactly one second as long as the following general criteria are met: the user finds what they want or need the majority of the time, the user interface is fairly easy-to-use and intuitive, that there are limited or no overbearing advertisement banners (whoops!), and that the page loads within a reasonable amount of time. I really haven’t done enough research to know what a ‘reasonable amount of time’ is, but I know from testing that very few sites today deliver true sub-second performance. So let’s tear into this a bit.

First don’t get me wrong, I think sub-second performance is a very worthy goal and I absolutely think every web developer worth his or her salt should strive as hard as they can to make their web page performance as fast as possible. Reality is that budgets can be limited, time frames for deliverables can be short and not everyone is a website performance expert. However…I think we need to start asking some hard questions to make sense of the one-second rule and understand how it can be applied in our everyday development work rather than simply taking it at face value, for example:

  • What industries in particular does the rule apply? I suspect it’s mostly applied to the online retail industry. I think web site visitors cut other industries a reasonable amount of slack.
  • Does the rule only apply to first time visitors?
  • Do repeat visitors abide by other page speed rules? Repeat visitors can take advantage of cached browser pages to speed up their viewing experience
  • How do lousy mobile internet connections factor into this equation? For example, if someone knows they have a lousy internet connection most of the time do they factor that into their online buying decisions?
  • Does the rule apply to all countries or just the U.S.?
  • What is the ideal internet connection speed that this rule is based on? It seems unlikely that a page would have to load under one second regardless of the connection speed.
  • Does this apply to only self-hosted websites? What if your website is hosted on Amazon Webstore or Etsy because with these sites you don’t really have any control over how their webservers, DNS, cloud or internet pipe are configured.

I then went about verifying who are the largest online retailers in the U.S. by sheer sales volume and I came up with Amazon, Staples, Apple and Walmart as good candidates for the top four. However you verify this list, we can all agree that these four sites generate a massive amount of internet traffic, billions of dollars in revenue per year and perhaps even a majority share of internet sales. Given the fact that these stores are were tens of millions of people successfully shop every day I wanted to use the seemingly undisputable shopping experience of these retailers as a basis for comparison.

It seems like a fair assumption that these retailers must be doing something right, and therefore whatever they are doing could be a potential guideline for others. I theorize people’s online shopping and surfing expectations are formed by the websites on which they shop the most. You tend to do in-store shopping at places where you are comfortable and that the same can be said for on-line shopping. Therefore, we need to understand these leading retailers performance baselines to get some basic numbers that we can compare our against our own websites performance.

The Device

For my device, I used my middle of the pack Nexus 4 on a DSL WiFi to ensure the best possible consistent connection. Where I live 4G speeds can fluctuate quite a bit during the day, so in order to normalize those issues out of the tests I simply went with WiFi:

Android Nexus 4, Android v4.4.2
Native Chrome browser
12 Mb/sec DSL/WiFi via G Band (verified between 10 – 12 Mb/sec) – Your own WiFi experience will vary significantly.

To measure performance, I used the latest desktop version of Chrome Canary and it’s new mobile inspection tools that were hooked up to my phone via USB cable. This works really, really well, by the way.

The Criteria

Here is what I was looking for. Your test results will vary based on your device, other applications running and internet connection speeds. I didn’t test iTunes because, well, I don’t use iTunes on my Android and it’s not a website. Believe it or not, when I went to Apple.com on my Android I got a desktop website and not a mobile website.

I chose the following criteria to put context around the very first page load, since that’s what Google seems to focus on the most. My goal was to load each page two times. First time is with an empty browser cache and second time is with the website cached in the browser. Then I repeat the tests multiple times to help account for any anomalies.

Here’s the criteria I looked  at:

  • Page lag with the browser cache empty. This represents a first time page visitor. By the way, I’m making a distinction between the technical time that Chrome Devtools reports the page has loaded and when the various parts and pieces within a web page finish spinning up first. This may result in a slight delay until you can actually start navigating around. This is a very subject number and it’s really hard to eyeball it accurately, but we’ve all experienced it. A web page can ‘appear’ to look like it has loaded however when you go to scroll the page down nothing may happen for a short period of time. I fully acknowledge that some of my perceived delays are due to the lag time looking back and forth between a timer and the web page which were right next to each other.
  • Page lag with web page cached. I report both the technical page load time and the perceived page load time. This represents a repeat visitor.
  • Total download time with no cache. This is the time in which all HTML, JavaScript and CSS has finished loading. As mentioned in bullet 1 this represents the number reported by the developer tools. Lazily loaded content can continue to happen unbeknownst the to user and drag down page performance for the first time page visitor.
  • Total download time with cache. Lazily loaded content can also drag down page performance for repeat visitors.
  • And last but not least, Google’s PageSpeed Insights online tool gives a few guidelines for trying to examine how well a website page stacks up against specific criteria. My only sticking point is it’s not 100% clear what criteria is being used. But, I will point out that not a single top four website got excellent ratings in the ‘speed’ category. In fact, if we were giving out grades, two of them were in the ‘C’ category and the other two were in the ‘F’ category.

Amazon.com Tests (Averaged)

  1. Page lag no cache – 1.18 seconds reported, however based on my perception it looked more like between 2 and 3 seconds as the page finished visually loading.
  2. Page lag cached – 1.53 seconds according to Chrome dev tools. Strangely, the cached page tests seemed just a hair slower than the non-cached page. I’ve noticed that browsers can sometimes be a bit slow when grabbing cached files. It would take more research to dig into how they construct their page and what cached settings are used.
  3. Total download time (no cache) 36.52 seconds, 819KB, 103 requests (Yes, that’s right…around 35 – 36 seconds for a full and complete page load)
  4. Total download time (cached) 4.59 seconds, 428KB, 78 requests
  5. PageSpeed Insights
    1. Speed – 71/100
    2. User Experience – 99/100

Walmart.com Tests (Averaged)

  1. Page lag no cache – 415ms reported, based on my perception it looked more like between 1 and 3 seconds as the page finished visually loading. There was a somewhat brief spinner icon that displayed as the page loaded. Fast!
  2. Page lag cached – 378ms actual, based on my perception it looked more like between 1 and 2 seconds. I was able to start scrollable immediately.
  3. Total download time (no cache) 10.32 seconds, 358KB, 41 requests
  4. Total download time (cached) 9.61 seconds, 47KB, 30 requests
  5. PageSpeed Insights
    1. Speed – 77/100 (This number surprised me, but again we don’t know how the number was calculated)
    2. User Experience – 99/100

Staples.com Tests (Averaged)

  1. Page lag no cache – 5.52 seconds reported, and that approximately matched what I could see.
  2. Page lag cached – 4.45 actual and that also matched what I could see.
  3. Total download time (no cache) 8.25 seconds, 572KB, 44 requests
  4. Total download time (cached) 7.04 seconds, 25KB, 39 requests. Wow, 7 seconds for 25KBs??
  5. PageSpeed Insights
    1. Speed – 50/100 (Yikes!)
    2. User Experience – 96/100

Apple.com Tests (Averaged)

Apple gets the worse grade of the group because when I surfed to www.apple.com I got a full blown desktop website instead of a mobile-enabled website. PageSpeed Insights apparently agreed with me.

  1. Page lag no cache – 2.41 seconds reported, and my eyeballing it said between 2 and 4 seconds.
  2. Page lag cached – 1.69 seconds according to Chrome dev tools. My eyeballing it tended to look like around 2 seconds.
  3. Total download time (no cache) 3.66 seconds, 2.8MBs, 72 requests.
  4. Total download time (cached) 2.46 seconds, 905KB, 71 requests.
  5. PageSpeed Insights
    1. Speed – 58/100 (Yikes!)
    2. User Experience – 60/100 (Double Yikes!)

Conclusions

Since the vast majority of internet users buy products from these major retails, I believe their overall perceptions of how a web site should perform is in a great part established by their experiences while buying products online from them. None of the top sites were perfect, and there is always room for continued improvements.

Only one website out of the top four Internet retailers delivered a technical page load speed that was under one second: Walmart.com. Amazon came really, really close. Staples had mediocre mobile performance. Apple didn’t offer my Android phone a mobile-enabled website.

There is a difference between the times when the page is loaded in the browser as reported by the developer tools and when all web page components become completely visible and then a short time later, ranging from several hundred to several thousand milliseconds, fully usable. As a mobile web developer I can tell you it takes a bit of time for a mobile application to be 100% ready. Many (most?) of us have experienced the often herky-jerky surfing experience as a web page bounces up and down while the content is still loading in the background. iPads have this nasty habit if you aren’t patient enough to wait and wait until the page “appears” to have finished loading. Because of this experience, defining technically at what point the page becomes fully usable can be fairly subjective. This is especially true because some retails treat tablets like desktop machines and deliver a full blown version of a website. Testing for when the page becomes visible and usable is very dependent on the phone’s capabilities, any other applications that might be using the phones hardware and bandwidth resources, as well as the internet connection at that point in time and the users perceptions!

Repeat page visits almost always load faster. Web developers would already know this, but it’s important to keep in mind for making sense of page performance discussions: first time visitors will get a different experience than repeat visitors who come back frequentyly. There is all sort of magic that can be done to control and tweak page caching.

References

Mobile Analysis in PageSpeed Insights
Mobile Path to Purchase: Five Key Findings (interesting info on how people use mobile for retail)
Amazon’s sales versus others (WSJ)
Top 5 largest online retailers (netonomy.net)

Tags: , ,
Posted in Performance | No Comments »

4 reasons user interface workflows are important

This blog post is for teams that are looking to build new applications, or are rebuilding existing systems from the ground-up. User interface workflows are the steps someone needs to take in order to complete a single task. These are no different from any other system you may have for doing things in your daily life. We all have systems. Sometimes we implement our own systems without even thinking about it because we they can make our lives intuitively easier.

Some common examples of systems that you might use are making coffee every morning and then drinking it on the way to work. Or, taking the same route to and from work every day. Eating dinner at the same place every Friday night. Maybe you have a system for creating passwords. The list goes on. We often also refer to these as rituals. It’s rare, or perhaps even unheard of, for us to devise a ritual for ourselves that causes frustration or anger. Eventually a ritual, or system, can become a habit and then you don’t even think about doing it anymore…it just happens.

User interfaces deserve the same love and care as any self-imposed ritual or system you’ve ever devised. We’ve all experienced bad user interfaces: things are hard to find, or finishing a task isn’t intuitive, or the application breaks. A good user interface follows a natural progression and it simply just flows along with a minimal number of instances where you simply “get stuck.”

The 4 Reasons

Note, I’ve vastly simplified these reasons and I could certainly write a lot more on each topic. So rather than blathering on and making this an academic-grade paper, I think these speak for themselves and can hopefully spark constructive conversation within your team or organization.

Time and money. Intuitive workflows are enjoyable to use and almost second nature. If a workflow is non-intuitive, it can cost your organization time and money as users struggle to complete tasks. Long term technical support costs can be a direct reflection of how easy or hard your application is to use.

Attracting/retaining customer. If you own an online retail site, where seconds matter to web visitors with the sub-second attention spans, having a well-designed site can actually help attract and retain customers. New customers appreciate when a system is easy-to-use. Systems that take to long to learn can cause customers to leave and never come back.

Training costs. Training costs can be lower for well-designed systems, especially if the concepts are easier to grasp. More complex systems, of course, have a longer learning curve and involve more in-depth training and training follow-up.

Modification resistance. There is typically some resistance to change and change is inevitable for most applications as time goes on.  Re-training costs are almost always taking into account when systems are modified in any way. If a system has been well perceived to begin with then small changes may not be a big deal. However, if a system has a perception of being very complex then there will be an expectation that any new additions will also be very complex. It may be fair to say within an organization that any resistance to change increases along with perceived complexity of the application.

What are some examples of simple workflows?

I believe travel web sites have nailed down their workflows pretty well and we can learn a lot of from them. Competition is fierce in the travel industry and they live and die on generating high sales volume. Sites such as Hotwire, Travelocity and others have to get you the information you need as fast and as easy as possible or visitors simply go elsewhere. Because of this pressure, they have to deliver exactly what you are looking whether it’s a simple round trip plane ticket, or if you need lodging, rental cars, an entire travel package or more.

How hard is it to build a simple user interface?

In many software circles there are often discussions proposing it’s far easier to quickly start coding an interface and to simply get the project going than it is to take the time and design a really good interface beforehand.  If you’ve ever worked with a user interface designer you’ll have had this concept strongly debunked and new concepts drilled into your brain. Understand and agree on the design first with all interested stake holders…well before anyone types a single line of code.

It can seem counter-intuitive at first to hear about spending time at the whiteboard drawing story boards and diagrams rather than simply sitting at your desk slamming down lines of code and pushing out prototype after prototype. I can guarantee one thing: It is far faster, easier and significantly less expensive to change an initial user interface storyboard in something like Basalmiq than it is to re-write user interface code over and over again as different groups try to agree on the best workflow.

Tags: , ,
Posted in UX | No Comments »

Calculate size of indexedDB database

IndexedDB has no built-in mechanism for calculating the size of a database. Below is a code snippet that shows how to calculate the ‘approximate’ size if you are storing strings. In my case we are storing Base64 images.

Depending on what data type you are storing here are some suggestions on how to calculate the size:

  • String. The example shown below simply uses the String.length property. You can typically get a one-to-one ratio where each character is equal to one byte. For example, if length equals 20, you can roughly assume that the size of the string is 20 bytes. However, if you have many uncommon characters in your strings those can equal 2 bytes each. For more information see this page on the Mozilla Developer Network.
  • Array. The size of an array depends on the array type and what you are storing in it. If it’s just a plain old Array of strings, such as ["a","test","something"],  you can parse the array for strings and then measure the size of each item within it as I just mentioned. If it’s a typed array you’ll have to take into account if it’s 8-bit, 16-bit, 32-bit or 64-bit. If you have numbers in your array, I believe all JavaScript numbers are 8 bytes each.
  • Blob. Access the Blob.size property.
/**
 * Provides the size of database in bytes
 * @param callback callback(size, null) or callback(null, error)
 */
var size = function(callback){
    if(this._db != null){
        var size = 0;

        var transaction = this._db.transaction(["your_db"])
            .objectStore("your_db")
            .openCursor();

        transaction.onsuccess = function(event){
            var cursor = event.target.result;
            if(cursor){
                var storedObject = cursor.value;
                var json = JSON.stringify(storedObject);
                size += json.length;
                cursor.continue();
            }
            else{
                callback(size,null);
            }
        }.bind(this);
        transaction.onerror = function(err){
            callback(null,err);
        }
    }
    else{
        callback(null,null);
    }
}

Usage

Here’s how you would use this in your application:

getSize(function(size,err){
     if(size > 0){
         console.log("Database size = " + size + " bytes");
     }
     if(err != null){
         console.log("There was a problem getting database size: " + err);
     }
}

Reference

W3C Specification for IndexedDB Database API
MDN – Blog

Tags: ,
Posted in JavaScript | No Comments »

I’ve lost count of the number of times I’ve been told by web development teams that a public facing web site must be okay because there haven’t been any (or many) complaints. True story. Okay…sure, that is one way to measure success. But now that I’ve called this out, most of you will likely agree this shouldn’t be the first metric you look at.

Websites need to be looked at through the lens of our users. I think people take things personally when they visit a website. Web surfers form expectations based on all the other sites they visit as they go about their day, it’s not just about your website. If you want them to come back often, they need to like using your website. It’s similar to anything else in life. If it’s frustrating to do something then you’re not likely to keep doing it over again for very long.

An excellent website that fosters long term loyalty will easily answer three questions:

  1. How easy is it to find what I want?
  2. How long did I have to wait for the pages to load?
  3. Did the website work on my particular device?

You may, in fact, have the most awesome, beautiful GPU intensive website ever built. However, if people can’t find what they want or if they have to wait too long to get it, or it just didn’t work on their device of choice then you’ve lost them. For example, if you are monitoring your website stats and notice a shift from iOS-based devices to Android, then you’d better make sure your website works on Chrome.

Certainly there are many, many other questions to be asked when building an excellent website and, of course, I’m oversimplifying things quite a bit. But the nice thing about these three questions is you can always swing the conversation back to them. Use can use these as a tuning fork to help build a solid foundation for success.

I think we can all take lessons from 3rd party travel websites. These websites operate on thin margins and they have to do only one thing to be successful: sell as many travel packages as possible. There is one thing that is absolutely consistent across the successful travel sites: the primary call to action is always in the upper left hand corner and it asks the three most important questions right up front. What’s your starting point? Where are you going? How do you want to get there? Bam.

Tags: ,
Posted in Innovation | No Comments »

Upgrade Anxiety – We all have it

As I type this I can see an icon indicating 10 plugins need to be upgraded on my blog. Most are minor upgrades with tweaks and fixes that don’t really affect me. Some, like the WP Super Cache update, are enough to give me nightmares of my blog having serious technical issues.

Oh wait…hah! Well, I actually did totally mess up my blog a couple years ago. I thought simply installing the WP Super Cache update would automatically preserve all of my settings. If you aren’t familiar with this WordPress plugin, it essentially provides a performance boost to your blog by having a gazillion caching-related knobs, bells and whistles that you can tune.

Yes, I did research the topic of upgrading beforehand. And at the time, deja vu, I found very little useful information other than people recommending you should upgrade. Caveat Emptor. I ended up paying a tech support person $175 to fix my mistake and my blog was totally messed up for about a week.

Okay, so now I make sure to backup/export all my WP Super Cache settings . Yet I still get the heebie-jeebies every single time I get its plug-in update notification.  A few friends offered some not-so-tongue-in-cheek suggestions of why don’t I just learn about all the different settings and just experiment. “You have the skill(z),” they told me. I couldn’t argue with that.

What it really comes down to is “how” I want to spend my time. Like most members of the Esri Developer Evangelist team, I’m totally slammed at work and the outdoors keep me busy after hours and on most weekends. I don’t really care to learn the nitty-gritty intricacies of WP Super Cache and it’s hundreds of configuration permutations, along with all the other stuff I have to learn to stay on top of the latest technologies, APIs, etc.

In hindsight, now I know the WP Super Cache website doesn’t have a single link or tab that explains the various settings and configuration options. There are ten steps listed that don’t even begin to cover what happens if you change something, or what are the pluses and minuses of doing one setting versus another.

Lessons Learned:

  • You don’t have to accept all upgrades whether it’s your laptop, smartphone app, etc. I’ve gotten really good at skipping some. Others, like Chrome, happen automatically and even though they also occasionally introduce new bugs.
  • Note to self: Always, always back up your software, databases and settings.
  • Some upgrades deserve more attention than others. WP Super Cache is one of those upgrades that deserves your full attention to details. Spend time on forums reading and asking questions before hitting the upgrade button.
  • Some upgrades simply aren’t worth it. I’ve dropped a number of plug-ins that mysteriously started gathering information from my blog such as AddThis. I was running a protocol analyzer at the time and I noticed strange http requests to a 3rd party URL. Not only were the synchronous http requests slowing down my website, but when I visited the 3rd party website the details of what they did were extremely vague.
  • The last lesson learned is actually a note to software vendors that may be reading this. Too many upgrade notifications per month is very annoying. Unless you find a fatal bug, one upgrade per month is about my limit.
Tags: ,
Posted in WordPress | No Comments »