Cross System Search

Here is an interesting idea. A search that can access content beyond multiple types of authentication. Now on the surface this is no more of a request than what any other search company is doing. You may have already heard of the quest for finding information buried in the "dark web". Kinda like the Dark Portal only much less sinister.

The problem is present in any organization that uses multiple authentication providers. While many corporations have a consistent technology structure there exists another entity that is a bit more liberal when it comes to choosing a tool for the job. Namely Universities. The answer here is use whatever you want as long as it does not cost any money. Here leads to the dilemma.

Now authentication can be fairly standard in an organization, most of the time. The authenticating mechanism may be accessing the same database (Active Directory or LDAP) but it may look different on the front end. Apache mod_auth_krb to IIS Integrated Authentication to Pubcookie. Now, on an intranet with all these flavors, how can you search all information?

Let's assume you can authenticate the user so any search agent can impersonate the user logged in at the time. How can you pass through these portals to get the information hidden behind. Sure, they are probably different systems doing different jobs, but they are all rendering HTML to the browser. There must be a way to get to this information without clicking through each of the systems independently.

SharePoint can do some of this by indexing certain sites, however, it can't get to anything protected by Pubcookie. So now what? I don't have any answers, but the person who does could make quite a bit of money a good name for him or herself by creating a agent that can perform this small miracle. So..... can it be done?