Network: Web Design: Finding an irritation bypass when in search of Marx

IT IS universally agreed that finding information on the Web can be damn irritating. It's not impossible, of course. If it was, no one would use it (and we'd be out of jobs).

Most people, both novice and experienced Net users, will start their quest for information with one of the many search engines. They type in a description of what they are seeking - say, a Marx Brothers movie clip from A Night at the Opera - so they type in "Marx Brothers", hit "search" and are whisked away to what is usually a massive list of possible Web pages that might, just might, have information that is relevant to them.

Naturally if you have that famous movie clip of Groucho trying to stuff the entire crew of the cruise ship into his cabin, then you want one of those websites placed towards the top, if not at the very top, of the search results list. Yet, you are potentially competing against thousands of other Web pages that have the key words "Marx" and "Brothers" on them, some of which might be extolling the virtues of the proletariat rather then containing the comedic genius.

So, how do you separate Karl Marx's philosophies from the Groucho Marx's shtick? Vladimir Lenin's revolution from John Lennon's "Revolution?" How do you make sure that the content you have gets found by the people who need it?

The place to start is by understanding how the search engines that your potential visitors will be using work.

Different flavours

Although the outcome is often the same - a list of search results - there are really two different types of search engines on the Web: crawlers and directories. These two methods differ primarily in the ways that they gather the data from which they create their index of sites, which is then searched.

Crawlers: Crawlers, such as AltaVista or Excite, use a program called a spider, which "crawls" through the Web, indexing pages along the way. Visitors can then search through the results that the spider finds. However, if a change is made to a Web page, the spider has to crawl through that page again before the change is detected. The World Wide Web is a really big place, so it might take a while for the spider to get back again.

Directories: Unlike the active crawlers, passive directories require website creators (or whoever wants to do it) to register a site in their index. The advantage of a directory such as Yahoo! or DMOZ, is that they are far more selective as to what content is indexed, so searches tend to be more focused and produce more accurate results. However, directories are also harder to keep up to date, especially if a site has to be checked by a human being before entry. The other great advantage of a directory is that the searcher can actually bypass the search engine, and find what they are looking for by narrowing down the subject by selecting from lists of increasingly specific topics.

Hybrid Search Engines: Several search engines, for example, Yahoo!, will allow you to search indexes created both by crawlers and directories simultaneously. This allows you to deploy the advantages of both techniques at the same time.

The parts of a search engine

Whether the search engine uses a crawler or a registration directory to get its data, they all have at least two parts in common: the index and the search software.

The Index: All of the content that gets crawled by the spider, and/or all of the entries in the directory get placed into the index. If the search engine uses a spider, then this massive database can contain every page that has been crawled, making it a carbon copy of the Web. If the search engine uses a directory, then only the titles, URLs, and descriptions of Web pages are included in the index.

The Search Software: When a visitor uses a search engine, they first enter one or more keywords. The index is then sifted through by search software which matches the key word(s) to Web pages and ranks them in order of relevance. So, how does the search software make the crucial decision as to which pages are more relevant, and thus closer to the top of the list, than others?

Ranking Web sites

Most search engines that use a crawler to produce the massive amounts of data used to search, determine relevancy by following a set of rules that stay more or less consistent across products. If someone is using the search engine to find the words "Marx Brothers", the search engine will check to see:

Which pages have these words in the

Which pages one or both of the words appear on.

How close to the top the words appear, assuming that the closer the words are to the beginning of a page the more relevant that page is.

How frequently the words appear on the page.

How close the words appear together.

And after considering all of these criteria, it produces the list of sites in order of relevancy. Well, almost.

Secret ingredients

While all of the major search engines follow this basic recipe, if all search engines worked exactly same way, then we would only need one search engine. Some crawlers index more pages than others, while many directories will use human beings to evaluate submitted websites. All search engines will put their own spin on searching to differentiate themselves from the competition.

Next week, I'll go further in depth into some of the secret ingredients that different search engines use to find your site. And then over the following weeks I'll be taking a look at how to optimise a site for searching, and some of the resources online to help you get a handle on the search engine monster.

Jason Cranford Teague is the author of 'DHTML For the World Wide Web'. If you have questions, you can find an archive of his column at Webbed Environments ( or e-mail him at

Arts and Entertainment
Cillian Murphy stars as Tommy Shelby in Peaky Blinders

Arts and Entertainment
The cast of The Big Bang Theory in a still from the show

Arts and Entertainment

Arts and Entertainment
Rupert Murdoch and Rebekah Brooks in 2011

Review: A panoramic account of the hacking scandal

Arts and Entertainment
Adèle Exarchopoulos and Léa Seydoux play teeneage lovers in the French erotic drama 'Blue Is The Warmest Colour' - The survey found four times as many women admitting to same-sex experiences than 20 years ago

Arts and Entertainment
Comedian Jack Dee has allegedly threatened to quit as chairman of long-running Radio 4 panel show 'I’m Sorry I Haven’t A Clue'

Edinburgh Festival
Arts and Entertainment
Director Paul Thomas Anderson (right) and his movie The Master featuring Joaquin Phoenix

Have you tried new the Independent Digital Edition apps?
Arts and Entertainment
Christian Grey cradles Ana in the Fifty Shades of Grey film

Arts and Entertainment
There are no plans to replace R Kelly at the event

Arts and Entertainment
Carmichael- Lady Edith Crawley</strong></p>
<p>Carmichael currently stars as Sonya in the West End production of
Chekhov’s Uncle Vanya at the Vaudeville Theatre. She made headlines this autumn
when Royal Shakespeare Company founder Sir Peter Hall shouted at her in a
half-sleepy state during her performance. </p>
<p>Carmichael made another appearance on the stage in 2011, playing
two characters in David Hare’s <em>Plent</em>y
at the Crucible Theatre in Sheffield. </p>
<p>Away from the stage she starred as receptionist Sal in the 2011
film <em>Tinker Tailor Solider Spy</em>. </p>

Arts and Entertainment

Arts and Entertainment
Zoe Saldana admits she's

Arts and Entertainment
'Old Fashioned' will be a different kind of love story to '50 Shades'
Arts and Entertainment

Arts and Entertainment
The Great British Bake Off contestants line-up behind Sue and Mel in the Bake Off tent

Arts and Entertainment
Mitch Winehouse is releasing a new album

Arts and Entertainment

Arts and Entertainment
Beast would strip to his underpants and take to the stage with a slogan scrawled on his bare chest whilst fans shouted “you fat bastard” at him

Arts and Entertainment
On set of the Secret Cinema's Back to the Future event

Arts and Entertainment
Gal Gadot as Wonder Woman

Arts and Entertainment
Pedro Pascal gives a weird look at the camera in the blooper reel

Arts and Entertainment
Public vote: Art Everywhere poster in a bus shelter featuring John Hoyland
Arts and Entertainment
Peter Griffin holds forth in The Simpsons Family Guy crossover episode

Arts and Entertainment
Judd Apatow’s make-it-up-as-you-go-along approach is ideal for comedies about stoners and slackers slouching towards adulthood
filmWith comedy film audiences shrinking, it’s time to move on
Arts and Entertainment
booksForget the Man Booker longlist, Literary Editor Katy Guest offers her alternative picks
Arts and Entertainment
Off set: Bab El Hara
tvTV series are being filmed outside the country, but the influence of the regime is still being felt
Arts and Entertainment
Red Bastard: Where self-realisation is delivered through monstrous clowning and audience interaction
Travel Shop
the manor
Up to 70% off luxury travel
on city breaks Find out more
Up to 70% off luxury travel
on chic beach resorts Find out more
sardina foodie
Up to 70% off luxury travel
on country retreats Find out more
Latest stories from i100
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    Dress the Gaza situation up all you like, but the truth hurts

    Robert Fisk on Gaza conflict

    Dress the situation up all you like, but the truth hurts
    Save the tiger: Tiger, tiger burning less brightly as numbers plummet

    Tiger, tiger burning less brightly

    When William Blake wrote his famous poem there were probably more than 100,000 tigers in the wild. These days they probably number around 3,200
    A tale of two presidents: George W Bush downs his paintbrush to pen father’s life story

    A tale of two presidents

    George W Bush downs his paintbrush to pen father’s life story
    Save the tiger: The day America’s love of backyard tigers led to a horrific bloodbath

    The day America’s love of backyard tigers led to a horrific bloodbath

    With only six per cent of the US population of these amazing big cats held in zoos, the Zanesville incident in 2011 was inevitable
    Samuel Beckett's biographer reveals secrets of the writer's time as a French Resistance spy

    How Samuel Beckett became a French Resistance spy

    As this year's Samuel Beckett festival opens in Enniskillen, James Knowlson, recalls how the Irish writer risked his life for liberty and narrowly escaped capture by the Gestapo
    We will remember them: relatives still honour those who fought in the Great War

    We will remember them

    Relatives still honour those who fought in the Great War
    Star Wars Episode VII is being shot on film - and now Kodak is launching a last-ditch bid to keep celluloid alive

    Kodak's last-ditch bid to keep celluloid alive

    Director J J Abrams and a few digital refuseniks shoot movies on film. Simon Usborne wonders what the fuss is about
    Once stilted and melodramatic, Hollywood is giving acting in video games a makeover

    Acting in video games gets a makeover

    David Crookes meets two of the genre's most popular voices
    Could our smartphones soon be diagnosing diseases via Health Kit and Google Fit?

    Could smartphones soon be diagnosing diseases?

    Health Kit and Google Fit have been described as "the beginning of a health revolution"
    Ryanair has turned on the 'charm offensive' but can we learn to love the cut-price carrier again?

    Can we learn to love Ryanair again?

    Four recent travellers give their verdicts on the carrier's improved customer service
    Billionaire founder of Spanx launches range of jeans that offers

    Spanx launches range of jeans

    The jeans come in two styles, multiple cuts and three washes and will go on sale in the UK in October
    10 best over-ear headphones

    Aural pleasure: 10 best over-ear headphones

    Listen to your favourite tracks with this selection, offering everything from lambskin earmuffs to stainless steel
    Commonwealth Games 2014: David Millar ready to serve up gold for his beloved Scotland in the end

    Commonwealth Games

    David Millar ready to serve up gold for his beloved Scotland in the end
    UCI Mountain Bike World Cup 2014: Downhill all the way to the top for the Atherton siblings

    UCI Mountain Bike World Cup

    Downhill all the way to the top for the Atherton siblings
    Save the tiger: The animals bred for bones on China’s tiger farms

    The animals bred for bones on China’s tiger farms

    The big cats kept in captivity to perform for paying audiences and then, when dead, their bodies used to fortify wine