Network: Web Design: Finding an irritation bypass when in search of Marx

IT IS universally agreed that finding information on the Web can be damn irritating. It's not impossible, of course. If it was, no one would use it (and we'd be out of jobs).

Most people, both novice and experienced Net users, will start their quest for information with one of the many search engines. They type in a description of what they are seeking - say, a Marx Brothers movie clip from A Night at the Opera - so they type in "Marx Brothers", hit "search" and are whisked away to what is usually a massive list of possible Web pages that might, just might, have information that is relevant to them.

Naturally if you have that famous movie clip of Groucho trying to stuff the entire crew of the cruise ship into his cabin, then you want one of those websites placed towards the top, if not at the very top, of the search results list. Yet, you are potentially competing against thousands of other Web pages that have the key words "Marx" and "Brothers" on them, some of which might be extolling the virtues of the proletariat rather then containing the comedic genius.

So, how do you separate Karl Marx's philosophies from the Groucho Marx's shtick? Vladimir Lenin's revolution from John Lennon's "Revolution?" How do you make sure that the content you have gets found by the people who need it?

The place to start is by understanding how the search engines that your potential visitors will be using work.

Different flavours

Although the outcome is often the same - a list of search results - there are really two different types of search engines on the Web: crawlers and directories. These two methods differ primarily in the ways that they gather the data from which they create their index of sites, which is then searched.

Crawlers: Crawlers, such as AltaVista or Excite, use a program called a spider, which "crawls" through the Web, indexing pages along the way. Visitors can then search through the results that the spider finds. However, if a change is made to a Web page, the spider has to crawl through that page again before the change is detected. The World Wide Web is a really big place, so it might take a while for the spider to get back again.

Directories: Unlike the active crawlers, passive directories require website creators (or whoever wants to do it) to register a site in their index. The advantage of a directory such as Yahoo! or DMOZ, is that they are far more selective as to what content is indexed, so searches tend to be more focused and produce more accurate results. However, directories are also harder to keep up to date, especially if a site has to be checked by a human being before entry. The other great advantage of a directory is that the searcher can actually bypass the search engine, and find what they are looking for by narrowing down the subject by selecting from lists of increasingly specific topics.

Hybrid Search Engines: Several search engines, for example, Yahoo!, will allow you to search indexes created both by crawlers and directories simultaneously. This allows you to deploy the advantages of both techniques at the same time.

The parts of a search engine

Whether the search engine uses a crawler or a registration directory to get its data, they all have at least two parts in common: the index and the search software.

The Index: All of the content that gets crawled by the spider, and/or all of the entries in the directory get placed into the index. If the search engine uses a spider, then this massive database can contain every page that has been crawled, making it a carbon copy of the Web. If the search engine uses a directory, then only the titles, URLs, and descriptions of Web pages are included in the index.

The Search Software: When a visitor uses a search engine, they first enter one or more keywords. The index is then sifted through by search software which matches the key word(s) to Web pages and ranks them in order of relevance. So, how does the search software make the crucial decision as to which pages are more relevant, and thus closer to the top of the list, than others?

Ranking Web sites

Most search engines that use a crawler to produce the massive amounts of data used to search, determine relevancy by following a set of rules that stay more or less consistent across products. If someone is using the search engine to find the words "Marx Brothers", the search engine will check to see:

Which pages have these words in the

Which pages one or both of the words appear on.

How close to the top the words appear, assuming that the closer the words are to the beginning of a page the more relevant that page is.

How frequently the words appear on the page.

How close the words appear together.

And after considering all of these criteria, it produces the list of sites in order of relevancy. Well, almost.

Secret ingredients

While all of the major search engines follow this basic recipe, if all search engines worked exactly same way, then we would only need one search engine. Some crawlers index more pages than others, while many directories will use human beings to evaluate submitted websites. All search engines will put their own spin on searching to differentiate themselves from the competition.

Next week, I'll go further in depth into some of the secret ingredients that different search engines use to find your site. And then over the following weeks I'll be taking a look at how to optimise a site for searching, and some of the resources online to help you get a handle on the search engine monster.

Jason Cranford Teague is the author of 'DHTML For the World Wide Web'. If you have questions, you can find an archive of his column at Webbed Environments ( or e-mail him at

Arts and Entertainment
Jamie Dornan as Christian Grey in the first-look Fifty Shades of Grey movie still

Arts and Entertainment
Sue Perkins and Mel Giedroyc, centre, are up for Best Female TV Comic for their presenting quips on The Great British Bake Off

Arts and Entertainment
Martin Freeman as Lester Nygaard in the TV adaptation of 'Fargo'

Arts and Entertainment
A scene from Shakespeare in Love at the Noel Coward Theatre
theatreReview: Shakespeare in Love has moments of sheer stage poetry mixed with effervescent fun
Have you tried new the Independent Digital Edition apps?
Arts and Entertainment
<p>Troubled actor Robert Downey Jr cements his comeback from drug problems by bagging the lead role in Iron Man. Two further films follow</p>

Arts and Entertainment

Arts and Entertainment
Tycoons' text: Warren Buffett and Bill Gates both cite John Brookes' 'Business Adventures' as their favourite book

Arts and Entertainment
Panic! In The Disco's Brendon Urie performs on stage

Arts and Entertainment

Arts and Entertainment
Keira Knightley and Benedict Cumberbatch star in the Alan Turing biopic The Imitation Game

Arts and Entertainment

Arts and Entertainment
Radio 4's Today programme host Evan Davis has been announced as the new face of Newsnight

Arts and Entertainment
Pharrell Williams performing on the Main Stage at the Wireless Festival in Finsbury Park, north London

Arts and Entertainment
Carrie Mathison returns to the field in the fourth season of Showtime's Homeland

Arts and Entertainment
Crowds soak up the atmosphere at Latitude Festival

Arts and Entertainment
Meyne Wyatt and Caren Pistorus arrive for the AACTA Aawrds in Sydney, Australia

Arts and Entertainment
Rick Astley's original music video for 'Never Gonna Give You Up' has been removed from YouTube

Arts and Entertainment
Quentin Blake's 'Artists on the beach'

Artists unveils new exhibition inspired by Hastings beach

Arts and Entertainment
MusicFans were left disappointed after technical issues
Arts and Entertainment
'Girl with a Pearl Earring' by Johannes Vermeer, c. 1665
artWhat is it about the period that so enthrals novelists?
Arts and Entertainment
Into the woods: The Merry Wives of Windsor at Petersfield
theatreOpen-air productions are the cue for better box-office receipts, new audiences, more interesting artistic challenges – and a picnic
Arts and Entertainment
James singer Tim Booth
latitude 2014
Arts and Entertainment
Lee says: 'I never, ever set out to offend, but it can be an accidental by-product'
Arts and Entertainment
tvThe judges were wowed by the actress' individual cooking style
Travel Shop
the manor
Up to 70% off luxury travel
on city breaks Find out more
Up to 70% off luxury travel
on chic beach resorts Find out more
sardina foodie
Up to 70% off luxury travel
on country retreats Find out more
Latest stories from i100
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    Screwing your way to the top? Good for Lana Del Rey for helping kill that myth

    Screwing your way to the top?

    Good for Lana Del Rey for helping kill that myth, says Grace Dent
    Will the young Britons fighting in Syria be allowed to return home and resume their lives?

    Will Britons fighting in Syria be able to resume their lives?

    Tony Blair's Terrorism Act 2006 has made it an offence to take part in military action abroad with a "political, ideological, religious or racial motive"
    Beyoncé poses as Rosie the Riveter, the wartime poster girl who became a feminist pin-up

    Beyoncé poses as Rosie the Riveter

    The wartime poster girl became the ultimate American symbol of female empowerment
    The quest to find the perfect pair of earphones: Are custom, 3D printed earbuds the solution?

    The quest to find the perfect pair of earphones

    Earphones don't fit properly, offer mediocre audio quality and can even be painful. So the quest to design the perfect pair is music to Seth Stevenson's ears
    US Army's shooting star: Lt-Col Steven Cole is the man Hollywood calls when it wants to borrow a tank or check a military uniform

    Meet the US Army's shooting star

    Lt-Col Steven Cole is the man Hollywood calls when it wants to borrow a tank or check a military uniform
    Climate change threatens to make the antarctic fur seal extinct

    Take a good look while you can

    How climate change could wipe out this seal
    Should emergency hospital weddings be made easier for the terminally ill?

    Farewell, my lovely

    Should emergency hospital weddings be made easier?
    Man Booker Prize 2014 longlist: Crowdfunded novel nominated for first time

    Crowdfunded novel nominated for Booker Prize

    Paul Kingsnorth's 'The Wake' is in contention for the prestigious award
    Vladimir Putin employs a full-time food taster to ensure his meals aren't poisoned

    Vladimir Putin employs a full-time food taster

    John Walsh salutes those brave souls who have, throughout history, put their knives on the line
    Tour de France effect brings Hollywood blockbusters to Yorkshire

    Tour de France effect brings Hollywood blockbusters to Yorkshire

    A $25m thriller starring Sam Worthington to be made in God's Own Country
    Will The Minerva Project - the first 'elite' American university to be launched in a century - change the face of higher learning?

    Will The Minerva Project change the face of higher learning?

    The university has no lecture halls, no debating societies, no sports teams and no fraternities. Instead, the 33 students who have made the cut at Minerva, will travel the world and change the face of higher learning
    The 10 best pedicure products

    Feet treat: 10 best pedicure products

    Bags packed and all prepped for holidays, but feet in a state? Get them flip-flop-ready with our pick of the items for a DIY treatment
    Commonwealth Games 2014: Great Scots! Planes and pipers welcome in Glasgow's Games

    Commonwealth Games 2014

    Great Scots! Planes and pipers welcome in Glasgow's Games
    Jack Pitt-Brooke: Manchester City and Patrick Vieira make the right stand on racism

    Jack Pitt-Brooke

    Manchester City and Patrick Vieira make the right stand on racism
    How Terry Newton tragedy made iron men seek help to tackle their psychological demons

    How Newton tragedy made iron men seek help to tackle their psychological demons

    Over a hundred rugby league players have contacted clinic to deal with mental challenges of game