Questions about /posts/ and others in robots.txt

Discussion in 'Forum SEO' started by Lpspider, Oct 19, 2018.

  1. Lpspider

    Lpspider Enthusiast

    145
    76
    +0
    I run Xenforo.

    I'm considering disallowing:

    /posts/
    /find-new/
    /members/

    but have not yet.

    Two questions:

    1. Should these be disallowed?

    2. What should I do if a large amount of threads with /posts/ in the URL are indexed by Google?

    Surely removing them will have a negative impact, at least short-term, for posts already indexed. They'll be deindexed, reducing any traffic they were bringing in, and possible GWT errors. On the other hand, can I test if they're being treated as duplicate content?

    What would you do? I realize blocking /posts/ is probably best practice and should have been implemented some time ago.

    3. What should I be looking at in GWT?
     
  2. we_are_borg

    we_are_borg Administrator

    5,339
    1,417
    +2,115
    1. I don't see any reason to disallow this.

    2. Are the links in Google valid does it go where it needs to go.
     
  3. Lpspider

    Lpspider Enthusiast

    145
    76
    +0
    They go where needed, yes. Mostly concerned with duplicate content.
     
  4. GrrasSolutions

    GrrasSolutions Aspirant

    39
    13
    +4
    It depends on you, which file you have to allow and what you have to Disallow, because you know what is important things of the website. and what files you should get crawled by search engine.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.