DARPA developing search engine for the murky deep web

11 Feb 2015

DARPA's headquarters. Image via Wikimedia Commons

The US military’s advanced research division – DARPA – is attempting to shine some light into the illegal activity contained within the internet’s lesser-known cousin, the ‘deep web’.

Having developed a program called Memex, DARPA have licensed the program out to 17 different developers for the last year that will eventually low the military, and organisations like the NSA, better find specific results for illegal activity, according to Wired.

However, the search engine will be many-times more powerful than the world’s most popular search engine, Google, and will search results across both the standard and deep web including the infamous Silk Road internet black market and even bypass the Tor encrypted browser.

Those concerned with what privacy Memex offers internet users may have issues with the search engine and its project lead, Dr Chris White, who when speaking of Memex says that it will do more than just find illegal activities, but also calculate hidden relationships between previously undiscovered websites and any links they may also have to illegal activities.

Google searches only a fraction of the internet

Giving a demo on CBS’s 60 Minutes news programme, Dr White gave the example of allowing police operations to determine who is posting adverts online for illegal sex trafficking and where they are located, which is then turned into a clearer data map which effectively gives a visual representation of trafficking across the world.

Of course, as it is still in development and wishing to keep its tech a secret, DARPA has not revealed any details of how Memex works, except that IP addresses come in to determining where they are, but is only a minor factor.

According to Dr White, only 5pc of the internet’s web pages are searched by Google which he suggests means there are thousands of undocumented web activity that remains hidden behind private servers.

This is not the first search engine to be developed for the deep web however as last April, a team of deep web users and enthusiasts decided to launch their own search engine – heavily-modelled off Google – known as Grams.

Colm Gorey was a senior journalist with Silicon Republic