%include.front.METADESIGN;
In 1990, while working under the tutelage of
the legendary Arlington Hewes at
The Phone Company (tpc.int),
we posited the need for Yet Another Protocol (YAP). At the
time, a variety of application protocols were under
active development, the web had not yet been fully born,
and the TCP protocol had not yet been revised to make
the port number a constant.
We immediately began work on the protocol, naming it
Blocks. We then deferred any future specification of
the details for 10 years in order to properly ponder
the problem.
The guiding metaphor of the Blocks protocol was the
map. The pain that produced the metaphor was network topology.
In those days, the Cisco router was not the marvel of
plug-and-play interoperability that we know today.
There were times when, through inattentive reading of the
step-by-step documentation, we screwed up the routes
and rendered our networks inoperable.
An old network management trick, when you screw up your
network cloud, is to telnet into the closest
router using the IP address (the DNS name being somewhat
useless if you can't see your DNS server). Once on that
first router, you telnet to the next one until,
after crawling over a series of data links, you
reach the offending misconfigured router, chant
a few incantations, and repair the net.
A map of the network topology seemed an obvious tool
in assisting such misguided operations (we leave aside
the obvious issue of the server that has your
map being in the unreachable part of your network).
Indeed, maps of other people's networks might even
be a useful tool for identifying services available
to the outside world, say an ftp, whois, or finger
server. Thus was born the idea of a protocol
that would assist in mapping the Internet, sharing
those maps, and giving people the ability to step
above the net and look around.
Many people regarded the SNMP protocol suite as
the mechanism that would allow these maps to be built by network
management software. In retrospect, SNMP had a much
more precise function. Rather than being the magic
bullet that reached the holy grail of visualizing
network topology on the fly, SNMP ended up being
a means of instrumenting network devices, allowing data
such as the number of packets dropped on a router or the
average power consumption of a UPS to be made
available to a network management agent.
The extensible nature of SNMP, through the use of
MIBs, made it a valuable model for protocol development.
The core protocol was fixed and simple, yet the MIB
effort allowed anyone and everyone to draft a
method of instrumenting devices from toasters to
satellites. If a monkey wanted to draft a banana tree
MIB, the SNMP protocol mechanism allowed the monkey
to do so, and it was up to the user base and the market
to decide if that MIB came into broad use. However,
the requirements of the banana MIB did not impact the
core protocol or even the development of other MIBs.
Unfortunately, SNMP did not solve the problem of divining
network topology. It was obviously a valuable source
of data, but any information gleaned from skulking
SNMP-accessible devices would have to be combined with
data from a wide variety of other resources such as
the routing tables. We thus moved from the issue
of visualizing the network to that of a flexible
mechanism for resource discovery.
The rich visual map of a network built from a large
number of resource discovery mechanisms was certainly
a tool for the professional network manager, but we
actually envisioned that this mechanism would prove
useful to end users who would want to see network
topology as a means towards better navigation.
We had a scenario why end users would care about network
topology, a notion that was considered somewhat
heretical by the "Internet hides all topology and
transparently connects end nodes" oral tradition
that serves as the Internet architecture. This
school of thought felt that networks should always
be transparent and that even things like domain names
would be hidden from the user.
Our scenario on why the end user should care ran
something like this. Let's say you engaged in an
MBONE video session with a user named Deering at
Xerox Parc. In 1990 and 1991, multicast was just
beginning the hypergrowth that has led to our
modern MBONE, and session directory protocols like SDP did not exist.
If the face at the other end of the video screen
says something interesting in a video conference,
your first inclination would be to look around the
subnetwork that is the source of the video. Is
there perhaps an FTP, mail, or finger server on
that subnet? Is there a little FTP server on the
same machine as the video? Most likely a personal
archive of documents. Perhaps a "big" server on
the same subnet (as evidenced by the number of
documents, size of machine, or kind of link)?
Perhaps the departmental server!
We thus saw maps of networks as a service location
and navigation tool. And, if the resource discovery
and map construction could be based on a well-defined
protocol, perhaps the effort of mapping the entire
Internet could be accomplished as a highly-distributed
enterprise. Indeed, such a protocol would allow
one group to map a network and then share that
map with other people. The key to these maps was
the distributed collection of data, the ability
to add to and personalize the data collected, and
the ability to construct and share different views
or interpretations of the underlying topology.
Maps are a metaphor, and one can argue that maps of
network topologies are the wrong metaphor to
be pursuing. After all, on the Internet, one
can build virtual worlds. Maps of topology had no
interest to many people, but building virtual
worlds attracted a huge following to efforts such
as VRML.
Our feeling was that if you were going to build
virtual worlds, you shouldn't start from scratch.
You should start with the real world (and to us
the mass of data and servers that is the Internet
is the real world) and use that as a bootstrap
mechanism, the raw materials that people would use to
build virtual worlds.
Network topology as a useful tool to the end
user and as the raw material of an effort
to construct virtual worlds became the elevator
pitch. Map became the metaphor.
In 1998 (after several unsuccessful attempts to spin up
the efforts), we formed Invisible Worlds and met
several times with our Protocol Advisory Board.
The Protocol Advisory Board provides advice and direction
on the core specifications for the Blocks Protocol Suite.
The Protocol Advisory Board is consensus driven, which means
its advice is not necessarily the product of, or agreed to by,
any particular member. Note that the participation of these people
on the Protocol Advisory Board does not consist of or imply an
endorsement by the member's respective employers, nor does their
participation consist of an endorsement in any of their various
official capacities. The members of the PAB at the time
included
David Clark, David Crocker, Paul Vixie, Paul Mockapetris, and
Steve Deering.
We
were determined to proceed on the production of an
Internet Atlas, a large-scale effort to map the
Internet.
But, what does mapping the Internet mean? A few people
bought the proposition that you started with a critical mass of
information from network topology and used this as
a bootstrap mechanism, but not everybody was convinced.
What became clear was that the map was the wrong metaphor.
And, because the map was the wrong metaphor, we were
solving the wrong problem.
A map of a network topology assumes there is something
to map. And, everybody is going to want to map
something different. The map assumes a space to be
mapped. Space is the proper metaphor and the
map is one possible visualization of that space. Our
first architectural principle thus became the late
binding of the collection of the data to the means of
visualizing that information.
Once we realized we were looking at a data flow
architecture with resources being discovered and
then visualized in a variety of ways, it became
equally clear that the problem we were dealing
with was a more general problem, the management of
metadata. Metadata defines a space and is the
raw material that one uses to navigate that space.
Space as a metaphor proved quite powerful, with
immediate applications to maps (or other navigation
means) not only of network topology, but of spaces
such as the web, or a particular collection of
related information.
Our job became one of
providing the user with what David Clark calls
the "up" button. Given a resource on the Internet,
say a document or a router, our job became that
of giving the user the ability to hit an up button,
take a step above, look around, and "see" what
resources are nearby. According to Clark, our
system should allow the user to define what
"near" means in any given context.
Our canonical example of a space became what we call
"deep wells" of information. Take the SEC's EDGAR
system as the beginning of a space.
EDGAR is a constant flow of filings by public corporations
that accumulates over time. The dimensions of the
space are pieces of metadata that are in common over
several of these filings, such as the name of the filing corporation, the company's state of incorporation, the form type, or the company's Standard Industrial Classification.
In this space, an annual report (known as a 10-K) by Cisco might be
"near" other objects on a variety of dimensions.
An earlier 10-K by Cisco, other filings by Cisco, a 10-K
by other companies in the same SIC code, or a 10-K in other
companies that Cisco has invested in are all objects that
are near the annual report in question.
The deep wells of information form the same bootstrap
mechanism that we had hoped to achieve with network
topology. The EDGAR database has several hundred
thousand documents that are rich in metadata.
One of the things we realized, however, is that
resource discovery and data mining are processes that
are difficult to do. There are many different
algorithms to use to discover things ranging from
simple transformations based on regular expressions to
complex linguistic analysis to determine the presence
of certain forms of business events (e.g., "any evidence of insolvency disclosed in this document").
A requirement for our application (and hence for
the underlying protocol supporting the application)
was that many different methods of resource discovery
had to be able to coexist. Spaces may be defined
through a simple process (e.g., taking each EDGAR
document and creating some metadata), but it must
also be possible for the spaces to accumulate over
time. In particular, we wanted many different resource
discovery mechanisms to coexist peacefully with each
other and with human beings.
It became clear that we needed to begin defining
some architectural principles. To illustrate why
this "location" and "navigation" protocol
called Blocks was needed, we'd been giving the
example of the pain caused by search
engines that returned 30,000 results on a simple
query. Why were the search engines not solving this
problem of navigation?
The modern search engine (indeed, any portal, vortal, or other buzzword denoting a large
amount of information served on-line) is in
a sense the classic centralized service. Crawling the
net for keywords, indexing and searching the
database of keywords, and preparing the results
as a web page are all bundled together in a single
proprietary, centralized solution (centralized in
the sense of the administrative boundaries, not in
the sense of the numbers of computers needed to
create any one service such as Yahoo).
While a Google, Yahoo, or Altavista are all complete solutions,
there is no interoperability among these services.
Only through cheap hacks can one ask a question of
Google and Altavista, and then combine the results
together. While there is no interoperability among
these services, it is clear that each of them has created
a space, a view of the underlying network.
The lack of interoperability among search engines and
portals was certainly one issue, but these solutions
also missed some of the things we considered important,
particularly programmatic access and late binding to
visualization. While one can perform cheap hacks to
programmatically access a Google, this is certainly
not a satisfactory solution (indeed, if you send too
many such programmatically-constructed queries, the
system is likely to start refusing them). And,
for the results returned, those are always in the
format that the portal decides is appropriate: a banner
ad, a few results, some formatting they decided looks
good.
We took the modern search engine and asked ourselves
what we could do to chop the monolithic application up
into several pieces, each specializing in a specific
task. Our solution came up with 3 pieces, each
focusing on a piece of the puzzle:
Finding things on the net, a process we call
mixing
Managing that metadata, the function of a traditional
server
Preparing that metadata for a particular application
and user interface, a process we call building
While the process of mixing can be achieved by a global
web crawler (indeed a global bot is a mixer), our
philosophy and hence the software we've built
focuses on more limited, specialized crawling inside
of deep wells and specifically targeted other
resources. While leading-edge, all-inclusive
algorithms to read every word on the Web are certainly
an honorable activity, we also wanted to make
sure that our architecture would leave room for
more specialized agents under the control of
domain experts. We wanted these specialized mixers
to be easy to make.
A mixer, ideally, should be able to exist with many
other mixers to create a space. Each mixer, focusing
on a few tasks of limited scope, contributes a set
of resources to the task. Since mixers can extract
metadata not only from the underlying network but
from the information produced by other mixers, the
process becomes incremental.
If today's big 5
search engines can only index 20-30% of the network,
our vision is of 1 million little mixers, each
examining 1%.
While the mixers are specialized modules, we left
out any specification of how the mixers find things.
These are implementation details. The server, on
the other hand, clearly needs to be in the middle of
an hourglass, with a very simple, fast and well-defined
interface to the mixers.
One of our goals with the server was to use
index and search techniques such as SQL databases and
full-text engines, software that has become a commodity.
Rather than writing our own, our goal was to have
the server use these commodities as the engine,
and to hide the details of any one commodity from
the mixer.
The mixer uses a very loose definition on one side of
the hour glass ("find things") and a very tight definition
of the interface to the server. Likewise, the
builder has a very clearly defined interface to the
server, and a very loose definition on the outside of
the hourglass. The job of the builder is to pipe data
into any user interface (or other output source).
The architectural philosophy of builders, mixers, and servers
has been expressed in an Architectural Precepts document,
in a core protocol (BXXP), and in a metadata application
called the Simple Exchange Profile. These building blocks
form the framework, but beg the issue of what to do with
that framework.
The architecture of mixers, builders, and servers seemed like a promising
one for chopping up the search engine functionality into a distributed
system. But, what language to use to describe the spaces?
Here, we entered a world that is very old yet highly immature. Schemas for the description
of spaces date back to X.500, and a variety of efforts throughout the years
have attempted to create the ultimate global directory.
For our spaces, we envisioned something a little different than the
X.500 concepts of country servers, state servers, city servers, and
institutional servers, all working together to format information
about our global population into one framework. Whereas X.500
organized the world in terms of geography and people, we saw spaces
as a much more abstract, flexible construct. In other words, we
needed a language for describing spaces that helped define a schema,
yet was schema agnostic enough to accommodate a wide variety of
different kinds of metadata.
While HTML didn't serve this purpose very well, it was quickly clear
that XML had emerged as the data description language for the
next millennium. XML has some properties that make it quite attractive.
First, the model of nested documents works well for our world of
objects that contained objects and relationships to other objects.
The XML committee's decision to simplify SGML, yet still support
proper characters through the use of the UTF-8 and UTF-16 subsets
of Unicode make XML a simple but very powerful language.
XML is the generic underpinning, a language for describing data. We
then looked at a variety of other XML-based initiatives to see if
they added power to our ability to describe spaces. The most
promising initiative is the Resource Description Framework.
RDF evolved out of the earlier PICS platform of the W3C, but
serves a much broader role than simply blocking out pornography sites.
Indeed, documents from the W3C explain that this framework
serves a large number of goals, including:
interoperability of metadata machine understandable semantics for metadata better precision in resource discovery than full text search future-proofing applications as schemas evolve a uniform query capability for resource discovery a processing rules language for automated decision-making about Web resources language for retrieving metadata from third parties
In addition, Tim Berners-Lee, in a technical note
further explains that metadata (and it's instantiation through the
RDF framework)
"will allow huge amounts of information in databases and
existing applications to be put on the web, not just for
human browsing but for machine understanding:
searching, reasoning, and analyzing."
Given these technical goals, it seemed to make sense to leverage
the RDF effort for our own application. We thus looked at a
variety of RDF specification documents and examples.
A typical example is the following by
Eric Miller:
The example illustrates a variety of concepts from the XML world.
First is the concept of namespaces,
defined by Bray et. al. for the W3C as a
"collection of names, identified by a URI reference" which provides
a mechanism for software to "recognize and act on these declarations
and prefixes." In other words, the namespace is a scoping mechanism.
In this example, there are two types of names: RDF names, and the Dublin
Core. The Dublin Core is an earlier mechanism for tagging metadata.
This example illustrates the concept of "triplets" on which RDF is
based (e.g., "URI" "about" "person"). The target here is a URI, the action is a "description" and
the Dublin Core Creator is John Smith. What is interesting is
the mixing of different schemas and schemes.
A further example serves to illustrate the mixing of schemes,
in this case based on the popular VCARD,
a digital business
card that is often attached to email messages:
Finally, we look at a third example, this one from the
Microsoft
BIZTALK framework:
One can argue that a VCARD or a BizTalk purchase order are not
metadata. However, it became clear to us that there would be
a variety of schemes advanced for the description of metadata
and that any mechanism we put into place should be agnostic
to those schemes, allowing space makers to use the mechanism
that fits most naturally into their particular space.
The Blocks architecture was previously defined in a series of Internet-Drafts
describing the core architecture, the BXXP application framework,
and
the Simple Exchange Profile. A conversational description of the
design rational for the BXXP application framework were also previously
described.
The protocol has been implemented as a series of 3 software modules that were then
applied to several "deep wells" of information, including the SEC's EDGAR databases.
The mixer software is implemented in the Tcl and PERL languages, the SpaceServer is
implemented in Tcl and uses Verity, Oracle and other commercial datastores to
store, index, and retrieve metadata. Finally, builders have been implemented in
Tcl with the most significant focus being on builders that act as a web proxy with
the Apache web server. Many of these modules have been described on the
developers site at http://mappa.mundi.net/.
The metadata framework we have designed explicitly left out the definition of several
key issues. In particular, we are schema and namespace agnostic, allowing a variety of
metadata models to be defined. In the particular case of EDGAR and other metadata
repositories we have used to test and develop software, no use of mechanisms such as
RDF have been directly employed.
The system of servers, mixers, and builders provide a distributed solution, but the
solution is one of "islands of distribution." It is up to mixers and space servers
to know the DNS address and port number of a particular server to communicate with.
As such, the servers have not been stitched together into a truly distributed,
coordinated global service.
The true distribution of a metadata service is a subject of the second part of
our architecture, known as the convergence model. The convergence model is used
for replication of metadata from one server to another, and is also the basis
for knowledge management and other metadata schema and discovery issues.
Finally, while we have put in hooks for namespace administration in the Blocks
protocol (see the Blocks eXtensible eXchange Service), we have also deferred further specification of those issues until more
operational experience has been gained. In particular, a mechanism for distributed
management of the namespace is dependent on the infrastructure for both knowledge
management and bulk replication. In the time-honored tradition of hosts.txt, we
thus manually administer the namespace until a better solution is necessary.
%include.reference.BXXA;
%include.reference.RFC.1528;
%include.reference.SNMP.ROSE;
%include.reference.RFC.2327;
%include.reference.W3C.RDFFAQ;
%include.reference.W3C.RDFARCH;
%include.reference.W3C.XMLNAMESPACES;
%include.reference.DLIB.RDF;
%include.reference.I-D.VCARD;
%include.reference.MICROSOFT.BIZTALK;
%include.reference.BXXP;
%include.reference.BSEP;
%include.reference.APPLDESIGN;
%include.reference.BXXS;