(NaturalNews) The Defense Department's most secretive research division has created a new computer program giving America's spies a powerful tool to search the so-called "dark web," where some of the most sophisticated terrorist organizations operate.

DARPA - the Defense Advanced Research Projects Agency - recently introduced Memex, which, according to the BBC, is "intended to search the Web in new and different ways."

Specifically, Memex is search engine software that is capable of spotting relationships and patterns in data.

According to a DARPA press release, the new software differs substantially from existing search engine software:

Today's web searches use a centralized, one-size-fits-all approach that searches the Internet with the same set of tools for all queries. While that model has been wildly successful commercially, it does not work well for many government use cases. For example, it still remains a largely manual process that does not save sessions, requires nearly exact input with one-at-a-time entry, and doesn't organize or aggregate results beyond a list of links. Moreover, common search practices miss information in the deep web - the parts of the web not indexed by standard commercial search engines - and ignore shared content across pages.

Memex will find what search engines currently cannot find

Memex overcomes those existing limitations with next-generation programming technology that will "revolutionize the discovery, organization and presentation" of all Web search results.

The agency says Memex will allow users to extend their current search capabilities, quickly allowing them to thoroughly organize subsets of information and data based on individual interests.

In addition, the program seeks to produce search results that are much more useful more quickly, as regards to specific domains and tasks. Programmers hope that Memex will "improve the ability" of government, military and even commercial institutions to locate and organize publicly available information on the Internet that is crucial to their mission, the press release said.

"We're envisioning a new paradigm for search that would tailor indexed content, search results and interface tools to individual users and specific subject areas, and not the other way around," said Chris White, DARPA program manager. "By inventing better methods for interacting with and sharing information, we want to improve search for everybody and individualize access to information. Ease of use for non-programmers is essential."

Eventually, DARPA says that Memex will apply to any public domain content, but initially the agency plans to use it primarily as a key Defense Department tool in fighting human trafficking.

"Human trafficking is a factor in many types of military, law enforcement and intelligence investigations and has a significant web presence to attract customers," says the press release. "The use of forums, chats, advertisements, job postings, hidden services, etc., continues to enable a growing industry of modern slavery. An index curated for the counter-trafficking domain, along with configurable interfaces for search and analysis, would enable new opportunities to uncover and defeat trafficking enterprises."

Memex, circa 1945

The "deep web" - which is also called the Deepnet, the Invisible Web and the Hidden Web, is a portion of the Internet that is not indexed by standard search engines. Some have said searching the Web today using standard search engines is like dragging a net across the surface of the ocean; you will snag a great deal, but there is much more deep below that is subsequently missed. It is in these deep recesses of the Web that criminal and terrorist enterprises often operate.

Memex is a tool to give the government's military and civilian spy agencies the power to identify such activity. In particular, the software will accomplish that by using three technical areas of interest: Domain-specific indexing; domain-specific search; and DoD-specified applications, DARPA said.

"The program plans to use commodity hardware and emphasize creating and leveraging open source technology and architecture," the agency said, in the release.

The program itself derives its name and inspiration from a hypothetical device that was described in a 1945 article for The Atlantic Monthly, by Vannevar Bush, then-director of the U.S. Office of Scientific Research and Development. The article, "As We May Think," describe Memex as a combination of "memory" and "index," capable of storing and automatically cross-referencing all of a user's books, records and other data.

Sources:

http://www.bbc.com/news/technology-31808104

http://www.darpa.mil/newsevents/releases/2014/02/09.aspx