Web Informant #180, 12 December 1999

      We all know the Web is too slow. Finding the most troublesome spots takes part detective, part engineer, and part just plain persistence. So a number of companies are doing something about at least measuring and reporting on these bottlenecks, in the hope of fixing or at least avoiding them.

      I had a chance to evaluate their offerings recently. Keynote Systems and Service Metrics (recently acquired by Exodus Communications) have for-fee service offerings exclusively, while Web Partner has both free and for-fee services. Mercury Interactive has both service offerings and testing products available.

      Both Keynote and Service Metrics claim similar methodologies. Each places a series of 100 or more "agents" or software monitors at various locations around the world, connected to particular Internet backbone and primary access providers. These agents send signals through these networks to a series of common Web destinations, such as eCommerce, general consumer and portal sites, as well to custom destinations specified for a fee by the site's owner. The agents calculate packet delays, overall latency and other measurements to get to these sites, and then send this information back to a central repository. The company in its reports then summarizes this.

      Keynote is very open about the location of its agents - with 64 in the US and another 27 located around the world. Service Metrics doesn't divulge where their agents are located. Keynote also offers free samples. Various Keynote reports are available on the company's Web site, and visitors can try to obtain a free analysis of their own Web site (although I was unable to do so). Service Metrics currently doesn't offer any free services, and has very low-res reproductions of its reports on its Web site. Mercury offers Topaz ActiveWatch, which monitors a site's business processes from ten different locations on Exodus' network, and can provide for fairly sophisticated scripting of how to test the workflow for these processes. I signed up for a free trial, received my first sales call within a few minutes, and began getting reports within a day - very responsive and impressive. Active Watch uses the same scripts that the other Mercury products use: once a customer develops a script (to navigate to a particular page on a Web site, execute a particular transaction, etc.), it can be used across the board in many different products and services.

      Keynote's Perspective service costs range from $295 to $995 a month. The higher end puts them out of the reach of the average sized Web operator, and is on par with the access line charges for a typical business T-1 circuit. They also offer a stripped-down single-city Lifeline service for $695 per year, although this isn't really a very effective price point. Service Metrics has more reasonable fees than Keynote, ranging from $295 to $495 per month. ActiveWatch costs $750 per month.

      There are two other offerings, one lower-cost and one higher- cost. Webpartner.com offers Secret Shopper Checkout that monitors the checkout page every 15 minutes of a specified storefront ($349/yr). They have a free service that monitors basic site availability and will e-mail you weekly status reports that I've used over the past few years. Manage.com offers a complete eCommerce package, starting at several tens of thousands of dollars, with in-depth monitoring of various processes.

      The tricky part for each of these companies is measuring the performance of complex sites. Consider the following:

• eCommerce means more than just being able to click on a few buttons. Keynote's measurements are misleading when it comes to eCommerce, because the total customer experience isn't measured. The trouble is more than just going to a few pages and placing the order: the overall eCommerce experience extends to e-mail confirmations of the order and how promptly the vendor actually delivers the goods to the customer. Mercury's offerings can measure these things, but you have to set it up properly.



• Dynamic elements of various Web pages, including banner ads, special coded links, and database content. These introduce delays and additional complexity for Web visitors. For example, on my site I have a button next to my Web Informant subscription page. The graphic for the button comes from my mailing list provider, eGroups. If the eGroups site is down, the graphic won't display properly, and people can't sign up for a subscription either. How should the performance of this page be measured - when all the page elements are visible? What if my site is performing quickly but eGroups is lagging - how should this be reported? Service Metrics claims to support dynamic page elements with its latest software release, called SourceTracker. But I couldn't find any other information about this product on the company's Web site.



• Aggregating data is dangerous, particularly on the Internet, where different service providers can account for varying populations of users. An outage at MAE-East affects tens of thousands of users. An outage at a small ISP may not affect any customers for a particular storefront. And the monitoring service may only check once an hour or once every 15 minutes, but the site being monitored may be down or experiencing difficulties in between these times. Mercury's products and services can be tailored to monitor at specific times and intervals.



• Data ownership. On the Internet, privacy is a big concern and many companies can be reluctant to have their data living on some third party's hard disk, however carefully protected it might be.

      Without careful scrutiny of the methods used to produce each vendor's reports, it is hard to say exactly what each company is measuring. If you are interested in purchasing one of these products, I urge you to examine the reports produced and understand the kind of measurements you are getting for your money. And good luck keeping your site up.

      Happy holidays, one and all.