Network: Web Design: Finding an irritation bypass when in search of Marx

IT IS universally agreed that finding information on the Web can be damn irritating. It's not impossible, of course. If it was, no one would use it (and we'd be out of jobs).

Most people, both novice and experienced Net users, will start their quest for information with one of the many search engines. They type in a description of what they are seeking - say, a Marx Brothers movie clip from A Night at the Opera - so they type in "Marx Brothers", hit "search" and are whisked away to what is usually a massive list of possible Web pages that might, just might, have information that is relevant to them.

Naturally if you have that famous movie clip of Groucho trying to stuff the entire crew of the cruise ship into his cabin, then you want one of those websites placed towards the top, if not at the very top, of the search results list. Yet, you are potentially competing against thousands of other Web pages that have the key words "Marx" and "Brothers" on them, some of which might be extolling the virtues of the proletariat rather then containing the comedic genius.

So, how do you separate Karl Marx's philosophies from the Groucho Marx's shtick? Vladimir Lenin's revolution from John Lennon's "Revolution?" How do you make sure that the content you have gets found by the people who need it?

The place to start is by understanding how the search engines that your potential visitors will be using work.

Different flavours

Although the outcome is often the same - a list of search results - there are really two different types of search engines on the Web: crawlers and directories. These two methods differ primarily in the ways that they gather the data from which they create their index of sites, which is then searched.

Crawlers: Crawlers, such as AltaVista or Excite, use a program called a spider, which "crawls" through the Web, indexing pages along the way. Visitors can then search through the results that the spider finds. However, if a change is made to a Web page, the spider has to crawl through that page again before the change is detected. The World Wide Web is a really big place, so it might take a while for the spider to get back again.

Directories: Unlike the active crawlers, passive directories require website creators (or whoever wants to do it) to register a site in their index. The advantage of a directory such as Yahoo! or DMOZ, is that they are far more selective as to what content is indexed, so searches tend to be more focused and produce more accurate results. However, directories are also harder to keep up to date, especially if a site has to be checked by a human being before entry. The other great advantage of a directory is that the searcher can actually bypass the search engine, and find what they are looking for by narrowing down the subject by selecting from lists of increasingly specific topics.

Hybrid Search Engines: Several search engines, for example, Yahoo!, will allow you to search indexes created both by crawlers and directories simultaneously. This allows you to deploy the advantages of both techniques at the same time.

The parts of a search engine

Whether the search engine uses a crawler or a registration directory to get its data, they all have at least two parts in common: the index and the search software.

The Index: All of the content that gets crawled by the spider, and/or all of the entries in the directory get placed into the index. If the search engine uses a spider, then this massive database can contain every page that has been crawled, making it a carbon copy of the Web. If the search engine uses a directory, then only the titles, URLs, and descriptions of Web pages are included in the index.

The Search Software: When a visitor uses a search engine, they first enter one or more keywords. The index is then sifted through by search software which matches the key word(s) to Web pages and ranks them in order of relevance. So, how does the search software make the crucial decision as to which pages are more relevant, and thus closer to the top of the list, than others?

Ranking Web sites

Most search engines that use a crawler to produce the massive amounts of data used to search, determine relevancy by following a set of rules that stay more or less consistent across products. If someone is using the search engine to find the words "Marx Brothers", the search engine will check to see:

Which pages have these words in the

Which pages one or both of the words appear on.

How close to the top the words appear, assuming that the closer the words are to the beginning of a page the more relevant that page is.

How frequently the words appear on the page.

How close the words appear together.

And after considering all of these criteria, it produces the list of sites in order of relevancy. Well, almost.

Secret ingredients

While all of the major search engines follow this basic recipe, if all search engines worked exactly same way, then we would only need one search engine. Some crawlers index more pages than others, while many directories will use human beings to evaluate submitted websites. All search engines will put their own spin on searching to differentiate themselves from the competition.

Next week, I'll go further in depth into some of the secret ingredients that different search engines use to find your site. And then over the following weeks I'll be taking a look at how to optimise a site for searching, and some of the resources online to help you get a handle on the search engine monster.

Jason Cranford Teague is the author of 'DHTML For the World Wide Web'. If you have questions, you can find an archive of his column at Webbed Environments ( or e-mail him at

Arts and Entertainment
'Banksy Does New York' Film - 2014

Art Somebody is going around telling people he's Banksy - but it isn't the street artist

Arts and Entertainment
Woody Allen and Placido Domingo will work together on Puccini's Schicchi


Arts and Entertainment
The sixteen celebrities taking part in The Jump 2015


Arts and Entertainment
British author Helen Macdonald, pictured with Costa book of the year, 'H is for Hawk'
booksPanel hail Helen Macdonald's 'brilliantly written, muscular prose' in memoir of a grief-stricken daughter who became obsessed with training a goshawk
Arts and Entertainment
Tom DeLonge has announced his departure from Blink-182

Arts and Entertainment
In the picture: Anthony LaPaglia and Martin Freeman in 'The Eichmann Show'

Arts and Entertainment
Anne Kirkbride and Bill Roache as Deirdre and Ken Barlow in Coronation Street

tvThe actress has died aged 60
Have you tried new the Independent Digital Edition apps?
Arts and Entertainment

ebooksNow available in paperback
Arts and Entertainment

Arts and Entertainment
Marianne Jean-Baptiste defends Joe Miller in Broadchurch series two

Arts and Entertainment
The frill of it all: Hattie Morahan in 'The Changeling'

Arts and Entertainment
Gillian Anderson and David Duchovny may reunite for The X Files

Arts and Entertainment
Jeremy Clarkson, left, and Richard Hammond upset the locals in South America
A young woman punched a police officer after attending a gig by US rapper Snoop Dogg
Arts and Entertainment
Reese Witherspoon starring in 'Wild'

It's hard not to warm to Reese Witherspoon's heroismfilm
Arts and Entertainment
Word up: Robbie Coltrane as dictionary guru Doctor Johnson in the classic sitcom Blackadder the Third

Arts and Entertainment
The Oscar nominations are due to be announced today

Oscars 2015
Arts and Entertainment
Hacked off: Maisie Williams in ‘Cyberbully’

Maisie Williams single-handedly rises to the challenge

Arts and Entertainment
Eddie Redmayne in The Theory of Everything and Benedict Cumberbatch in The Imitation Game are both nominated at the Bafta Film Awards
Arts and Entertainment

Academy criticised after no non-white actors nominated

Arts and Entertainment
Damian Lewis shooting a scene as Henry VIII in Wolf Hall

Arts and Entertainment
A history of violence: ‘Angry, White and Proud’ looked at the rise of far-right groups


An expose of hooliganism masquerading as an ideological battle

Arts and Entertainment


Lee Hadwin can't draw when he's awake, but by night he's an artist

Arts and Entertainment


Arts and Entertainment
Michael Keaton in the 1998 Beetlejuice original


Arts and Entertainment
Olivia Colman and David Tennant star in 'Broadchurch'


Latest stories from i100
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    Isis hostage crisis: Militant group stands strong as its numerous enemies fail to find a common plan to defeat it

    Isis stands strong as its numerous enemies fail to find a common plan to defeat it

    The jihadis are being squeezed militarily and economically, but there is no sign of an implosion, says Patrick Cockburn
    Virtual reality thrusts viewers into the frontline of global events - and puts film-goers at the heart of the action

    Virtual reality: Seeing is believing

    Virtual reality thrusts viewers into the frontline of global events - and puts film-goers at the heart of the action
    Homeless Veterans appeal: MP says Coalition ‘not doing enough’

    Homeless Veterans appeal

    MP says Coalition ‘not doing enough’ to help
    Has The Archers lost the plot with it's spicy storylines?

    Has The Archers lost the plot?

    A growing number of listeners are voicing their discontent over the rural soap's spicy storylines; so loudly that even the BBC's director-general seems worried, says Simon Kelner
    Human skull discovery in Israel proves humans lived side-by-side with Neanderthals

    Human skull discovery in Israel proves humans lived side-by-side with Neanderthals

    Scientists unearthed the cranial fragments from Manot Cave in West Galilee
    World War Z author Max Brooks honours WW1's Harlem Hellfighters in new graphic novel

    Max Brooks honours Harlem Hellfighters

    The author talks about race, legacy and his Will Smith film option to Tim Walker
    Why the league system no longer measures up

    League system no longer measures up

    Jon Coles, former head of standards at the Department of Education, used to be in charge of school performance rankings. He explains how he would reform the system
    Valentine's Day cards: 5 best online card shops

    Don't leave it to the petrol station: The best online card shops for Valentine's Day

    Can't find a card you like on the high street? Try one of these sites for individual, personalised options, whatever your taste
    Greece elections: In times like these, the EU has far more dangerous adversaries than Syriza

    Greece elections

    In times like these, the EU has far more dangerous adversaries than Syriza, says Patrick Cockburn
    Holocaust Memorial Day: Nazi victims remembered as spectre of prejudice reappears

    Holocaust Memorial Day

    Nazi victims remembered as spectre of prejudice reappears over Europe
    Fortitude and the Arctic attraction: Our fascination with the last great wilderness

    Magnetic north

    The Arctic has always exerted a pull, from Greek myth to new thriller Fortitude. Gerard Gilbert considers what's behind our fascination with the last great wilderness
    Homeless Veterans appeal: Homeless in Wales can find inspiration from Daniel’s story

    Homeless Veterans appeal

    Homeless in Wales can find inspiration from Daniel’s story
    Front National family feud? Marine Le Pen and her relatives clash over French far-right party's response to Paris terror attacks

    Front National family feud?

    Marine Le Pen and her relatives clash over French far-right party's response to Paris terror attacks
    Pot of gold: tasting the world’s most expensive tea

    Pot of gold

    Tasting the world’s most expensive tea
    10 best wildlife-watching experiences: From hen harriers to porpoises

    From hen harriers to porpoises: 10 best wildlife-watching experiences

    While many of Britain's birds have flown south for the winter, it's still a great time to get outside for a spot of twitching