Is your forum searchable by spiders?

Discussion in 'Forum SEO' started by jilly, Feb 27, 2004.

  1. jilly

    jilly Fan

    929
    550
    +2
    I have my board set to not be searched by spiders, because the board is a women's support group, and they would not want the personal type things they talk about to show up in a google search on our topic or their username, if someone knew them and tried searching by their 'nickname'.

    However I believe that this probably makes the board membership grow much slower than it would. It's a toss up between making it harder for the board to be found, and keeping the community more private..My main website that the boards are associated with is indexed, just not the posts on the board.

    I was just curious about other boards:

    1. How long has your board been up?
    2. How many members do you have?
    3. Is your board indexed by search engines?

    For me
    1. april 2001
    2. 6000
    3. No
     
  2. wizard1974uk

    wizard1974uk Tazmanian Gremlin

    5,765
    790
    +21
    1. October 2003
    2. 33
    3. Yes
     
  3. Jon

    Jon Enthusiast

    136
    28
    +1
    It's been up Since October 03, It has 53 members, (so far) and can be indexed by spiders, in fact you'd be suprised the wierd stuff my board is indexed in the top 30 at Google for. :eek:
     
  4. Ras Masta

    Ras Masta Participant

    70
    11
    +0
    My board had been up since Jan 2001 before the crash
    It had 1581 Members
    Yes it was indexed.
     
  5. Ben

    Ben Participant

    62
    56
    +0
    May 2003
    212
    yes
     
  6. Quiet Anders

    Quiet Anders Adherent

    250
    58
    +0
    10 days
    6
    Don't know (where do I set this option in vB3?)
     
  7. Caislander

    Caislander Adherent

    330
    113
    +0
    10 april 02
    8000
    yes
    ps: quiet anders, generally this opition is setup in your robots.txt file
     
  8. Ted S

    Ted S Tazmanian Master

    6,127
    735
    +23
    1. Mid 2000
    2. 22,000
    3. For sure

    One important thing to remember about search engine indexing is that by default most forum software does not work well with bots. Yes, robots files are vital to being indexed well but they are not the thing to worry about! While google will crawl your php urls a bit they really don't handle anything with a session or unique key well, if you want to really get good rankings find some hacks to make your dynamic php urls APPEAR to be static .html urls. For vBulletin this is very easy and something I recommend all forums do for their showthread and forumdisplay links if nothing else. The more the engines can index the more obscure queries your board will show up for...
     
  9. Anonymous

    Anonymous Habitué

    1,319
    677
    +357
    Lakecs.com
    Feb 2003
    163
    Yes

    RiderInfo.com
    Feb 8, 2004
    54
    yes

    I brought lakecs.com online last with vB2 Feb and spend the most effort on making the site search engine spider friendly. This site is not promoted except for local word of mouth.

    I brought Riderinfo.com online jan 2004 with Deluxe Portal and had problems. We closed it and started from scratch and lost 37 members. We came back online with vB3 and have built back up. vB3 seems spider friendly out of the box. This site is promoted with go click.
     
  10. CyanSmoker

    CyanSmoker Beanie and Shades

    356
    143
    +0
    Y-F / Militate.com:
    1. I think it's going to be exactly one year...
    2. 3626
    3. Yes
     
  11. AngelAmidala

    AngelAmidala Adherent

    264
    58
    +0
    Christian Forums


    1) Started November 2001
    2) 27,782
    3) Yes and No

    Everything used to be spidered...but several members were concerned with their information being able to be seen by non-members and people doing searches on places like Google. So we made use of the "hide all information from non-members" feature which keeps non-members from viewing your profile as well as hiding all your threads, posts, etc. from Google. If members enable that, they don't get spidered. (Though there was a bug in this feature and it spidered people anyway. Bug was found and fixed. ;) )
     
  12. Elbonian

    Elbonian Aspirant

    36
    11
    +0
    Non-Christian Forums :D

    1) Started in June, 2000 (with UBB)
    2) 11,192 registrations (lots of "never returns," though).
    Busy hour tends to be 300-450 simultaneous users.
    3) Yes (but no "hacks" to increase indexing).

    One thing to note about allowing spiders is that they will run up your bandwidth usage. I can see bursts of traffic in our MRTG graph where the spiders seem to be retrieving stuff. If indexing isn't valuable to you, and if you are paying for extra bytes transferred, then you need to change your robots.txt file to exclude spiders in order to cut down your bandwidth costs.
     
  13. immotive

    immotive fred

    406
    130
    +1
    Nov 2002
    400
    Yes, google alone consume more than 500MB of bandwidth per month
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.