Network: Web Design: Finding an irritation bypass when in search of Marx

IT IS universally agreed that finding information on the Web can be damn irritating. It's not impossible, of course. If it was, no one would use it (and we'd be out of jobs).

Most people, both novice and experienced Net users, will start their quest for information with one of the many search engines. They type in a description of what they are seeking - say, a Marx Brothers movie clip from A Night at the Opera - so they type in "Marx Brothers", hit "search" and are whisked away to what is usually a massive list of possible Web pages that might, just might, have information that is relevant to them.

Naturally if you have that famous movie clip of Groucho trying to stuff the entire crew of the cruise ship into his cabin, then you want one of those websites placed towards the top, if not at the very top, of the search results list. Yet, you are potentially competing against thousands of other Web pages that have the key words "Marx" and "Brothers" on them, some of which might be extolling the virtues of the proletariat rather then containing the comedic genius.

So, how do you separate Karl Marx's philosophies from the Groucho Marx's shtick? Vladimir Lenin's revolution from John Lennon's "Revolution?" How do you make sure that the content you have gets found by the people who need it?

The place to start is by understanding how the search engines that your potential visitors will be using work.

Different flavours

Although the outcome is often the same - a list of search results - there are really two different types of search engines on the Web: crawlers and directories. These two methods differ primarily in the ways that they gather the data from which they create their index of sites, which is then searched.

Crawlers: Crawlers, such as AltaVista or Excite, use a program called a spider, which "crawls" through the Web, indexing pages along the way. Visitors can then search through the results that the spider finds. However, if a change is made to a Web page, the spider has to crawl through that page again before the change is detected. The World Wide Web is a really big place, so it might take a while for the spider to get back again.

Directories: Unlike the active crawlers, passive directories require website creators (or whoever wants to do it) to register a site in their index. The advantage of a directory such as Yahoo! or DMOZ, is that they are far more selective as to what content is indexed, so searches tend to be more focused and produce more accurate results. However, directories are also harder to keep up to date, especially if a site has to be checked by a human being before entry. The other great advantage of a directory is that the searcher can actually bypass the search engine, and find what they are looking for by narrowing down the subject by selecting from lists of increasingly specific topics.

Hybrid Search Engines: Several search engines, for example, Yahoo!, will allow you to search indexes created both by crawlers and directories simultaneously. This allows you to deploy the advantages of both techniques at the same time.

The parts of a search engine

Whether the search engine uses a crawler or a registration directory to get its data, they all have at least two parts in common: the index and the search software.

The Index: All of the content that gets crawled by the spider, and/or all of the entries in the directory get placed into the index. If the search engine uses a spider, then this massive database can contain every page that has been crawled, making it a carbon copy of the Web. If the search engine uses a directory, then only the titles, URLs, and descriptions of Web pages are included in the index.

The Search Software: When a visitor uses a search engine, they first enter one or more keywords. The index is then sifted through by search software which matches the key word(s) to Web pages and ranks them in order of relevance. So, how does the search software make the crucial decision as to which pages are more relevant, and thus closer to the top of the list, than others?

Ranking Web sites

Most search engines that use a crawler to produce the massive amounts of data used to search, determine relevancy by following a set of rules that stay more or less consistent across products. If someone is using the search engine to find the words "Marx Brothers", the search engine will check to see:

Which pages have these words in the

Which pages one or both of the words appear on.

How close to the top the words appear, assuming that the closer the words are to the beginning of a page the more relevant that page is.

How frequently the words appear on the page.

How close the words appear together.

And after considering all of these criteria, it produces the list of sites in order of relevancy. Well, almost.

Secret ingredients

While all of the major search engines follow this basic recipe, if all search engines worked exactly same way, then we would only need one search engine. Some crawlers index more pages than others, while many directories will use human beings to evaluate submitted websites. All search engines will put their own spin on searching to differentiate themselves from the competition.

Next week, I'll go further in depth into some of the secret ingredients that different search engines use to find your site. And then over the following weeks I'll be taking a look at how to optimise a site for searching, and some of the resources online to help you get a handle on the search engine monster.

Jason Cranford Teague is the author of 'DHTML For the World Wide Web'. If you have questions, you can find an archive of his column at Webbed Environments (www.webbedenvironments.com) or e-mail him at jason@webbedenvironments.com

PROMOTED VIDEO
Have you tried new the Independent Digital Edition apps?
Arts and Entertainment

books
Arts and Entertainment
Tim Vine has won the funniest joke award at the Edinburgh Festival 2014

Edinburgh
Arts and Entertainment
Peter Capaldi and Chris Addison star in political comedy The Thick of IT

TV
Arts and Entertainment
Judy Murray said she

TV
Arts and Entertainment
Tim Vine has won the funniest joke award at the Edinburgh Festival 2014

edinburgh
Arts and Entertainment
Jeremy Paxman has admitted he is a 'one-nation Tory' and complained that Newsnight is made by idealistic '13-year-olds' who foolishly think they can 'change the world'.

Edinburgh
Arts and Entertainment
Seoul singer G-Dragon could lead the invasion as South Korea has its sights set on Western markets
music
Arts and Entertainment
Gary Lineker at the UK Premiere of 'The Hunger Games: Catching Fire'
tv
Arts and Entertainment
Christian Bale as Batman in a scene from
film
Arts and Entertainment
Johhny Cash in 1969
musicDyess Colony, where singer grew up in Depression-era Arkansas, opens to the public
Arts and Entertainment
Army dreamers: Randy Couture, Sylvester Stallone, Dolph Lundgren and Jason Statham
film
Arts and Entertainment
The Great British Bake Off 2014 contestants
tvReview: It's not going to set the comedy world alight but it's a gentle evening watch
Arts and Entertainment
Umar Ahmed and Kiran Sonia Sawar in ‘My Name Is...’
Theatre
Arts and Entertainment
This year's Big Brother champion Helen Wood
arts + ents
Arts and Entertainment
Full company in Ustinov's Studio's Bad Jews
Theatre
Arts and Entertainment
Harari Guido photographed Kate Bush over the course of 11 years
Music
Arts and Entertainment
Reviews have not been good for Jonathan Liebesman’s take on the much loved eighties cartoon
Film

A The film has amassed an estimated $28.7 million in its opening weekend

Arts and Entertainment
Untwitterably yours: Singer Morrissey has said he doesn't have a twitter account
Music

A statement was published on his fansite, True To You, following release of new album

Arts and Entertainment
Full throttle: Philip Seymour Hoffman and John Turturro in God's Pocket
film
Arts and Entertainment
Kylie Minogue is expected to return to Neighbours for thirtieth anniversary special
tv
Independent
Travel Shop
the manor
Up to 70% off luxury travel
on city breaks Find out more
santorini
Up to 70% off luxury travel
on chic beach resorts Find out more
sardina foodie
Up to 70% off luxury travel
on country retreats Find out more
Have you tried new the Independent Digital Edition apps?

ES Rentals

    Independent Dating
    and  

    By clicking 'Search' you
    are agreeing to our
    Terms of Use.

    Ferguson: In the heartlands of America, a descent into madness

    A descent into madness in America's heartlands

    David Usborne arrived in Ferguson, Missouri to be greeted by a scene more redolent of Gaza and Afghanistan
    BBC’s filming of raid at Sir Cliff’s home ‘may be result of corruption’

    BBC faces corruption allegation over its Sir Cliff police raid coverage

    Reporter’s relationship with police under scrutiny as DG is summoned by MPs to explain extensive live broadcast of swoop on singer’s home
    Lauded therapist Harley Mille still in limbo as battle to stay in Britain drags on

    Lauded therapist still in limbo as battle to stay in Britain drags on

    Australian Harley Miller is as frustrated by court delays as she is with the idiosyncrasies of immigration law
    Lewis Fry Richardson's weather forecasts changed the world. But could his predictions of war do the same?

    Lewis Fry Richardson's weather forecasts changed the world...

    But could his predictions of war do the same?
    Kate Bush asks fans not to take photos at her London gigs: 'I want to have contact with the audience, not iPhones'

    'I want to have contact with the audience, not iPhones'

    Kate Bush asks fans not to take photos at her London gigs
    Under-35s have rated gardening in their top five favourite leisure activities, but why?

    Young at hort

    Under-35s have rated gardening in their top five favourite leisure activities. But why are so many people are swapping sweaty clubs for leafy shrubs?
    Tim Vine, winner of the Funniest Joke of the Fringe award: 'making a quip as funny as possible is an art'

    Beyond a joke

    Tim Vine, winner of the Funniest Joke of the Fringe award, has nigh-on 200 in his act. So how are they conceived?
    The late Peter O'Toole shines in 'Katherine of Alexandria' despite illness

    The late Peter O'Toole shines in 'Katherine of Alexandria' despite illness

    Sadly though, the Lawrence of Arabia star is not around to lend his own critique
    Wicken Fen in Cambridgeshire: The joy of camping in a wetland nature reserve and sleeping under the stars

    A wild night out

    Wicken Fen in Cambridgeshire offers a rare chance to camp in a wetland nature reserve
    Comic Sans for Cancer exhibition: It’s the font that’s openly ridiculed for its jaunty style, but figures of fun have their fans

    Comic Sans for Cancer exhibition

    It’s the font that’s openly ridiculed for its jaunty style, but figures of fun have their fans
    Besiktas vs Arsenal: Five things we learnt from the Champions League first-leg tie

    Besiktas vs Arsenal

    Five things we learnt from the Champions League first-leg tie
    Rory McIlroy a smash hit on the US talk show circuit

    Rory McIlroy a smash hit on the US talk show circuit

    As the Northern Irishman prepares for the Barclays, he finds time to appear on TV in the States, where he’s now such a global superstar that he needs no introduction
    Boy racer Max Verstappen stays relaxed over step up to Formula One

    Boy racer Max Verstappen stays relaxed over step up to F1

    The 16-year-old will become the sport’s youngest-ever driver when he makes his debut for Toro Rosso next season
    Fear brings the enemies of Isis together at last

    Fear brings the enemies of Isis together at last

    But belated attempts to unite will be to no avail if the Sunni caliphate remains strong in Syria, says Patrick Cockburn
    Charlie Gilmour: 'I wondered if I would end up killing myself in jail'

    Charlie Gilmour: 'I wondered if I'd end up killing myself in jail'

    Following last week's report on prison suicides, the former inmate asks how much progress we have made in the 50 years since the abolition of capital punishment