Thanks to a post from NewsForge on Open source search technology goes beyond keywords, I was directed to a description of the Semantic Indexing Project at Middlebury College. Aaron Coburn, the lead developer of the project, says his team is currently documenting its open source search toolkit and finishing up a new desktop search application that should be released later this month. From the project Web site:
The National Institute for Technology in Liberal Education (NITLE) and Middlebury College have been experimenting with algorithms to help unstructured data organize itself into conceptually useful categories without human intervention. Part of our motivation is to find an alternative to spending prohibitive amounts of time and money on marking up course materials, documents, and online collections with metadata by hand. For many of the most common markup standards in use today, such as SCORM or Dublin Core, it can actually take longer to create markup than it did to create the course materials themselves.
The method being applied is a more scalable variant of latent semantic indexing that the team calls contextual network graphing. A PDF paper from the project, Semantic Search of Unstructured Data using Contextual Network Graphs by Maciej Ceglowski, Aaron Coburn and John Cuadrado explains this promising technique in greater detail and notes its debt to a 1981 Ph.D. dissertation by Scott Preece at the University of Illinois describing an almost identical technique under the name spreading activation search.
The Semantic Indexing Project is an umbrella effort over a number of subsidiary projects including a blog census, literary analysis tool, refinement of search and clustering algorithms, bioinformatics, use of ontologies, and semantic relationship visualization through a Semantic Explorer, as this example shows:
All of the source code is available for download from the project, published under the terms of the GNU General Public License. The project’s core technology is the Semantic Engine, which is distributed with its C++ code, Perl bindings, and all the necessary code for building the GUI. A new desktop application, called the the Standalone Engine, will be available later this month.
This work looks very, very promising as a step forward to bringing automation to semantic Web markup, among related advantages deriving from tagged documents.
