question

DiegoLpez-2745 avatar image
0 Votes"
DiegoLpez-2745 asked Bruce-SqlWork answered

How to fix robots.txt file errors

Hello community how are you?

I am currently new with robots.txt files and SEO improvement, I am currently supporting an ASP.Net application and I am in the task of improving this feature of SEO positioning.

In what I was revisano with the lighthouse tool tells me that it will create a robots.txt file and add it to my project to improve the positioning, so I dedicated myself to that task and configure and create that file. What happens is that when I look at the lighthouse again, the SEO score has indeed gone up, but I keep getting robots.txt errors which tell me that it is because of my html, which doesn't make much sense to me. Next I show you my robots.txt file and the error that appears in lighthouse.

robots.txt file:

user-agent:
Disallow: /
.aspx
Disallow: /_controls/
Disallow: /_master/
Disallow: /Account/
Disallow: /Admin/
Disallow: /Error/
Disallow: /Login/
Disallow: /Public/

User-agent: Googlebot
User-agent: AdsBot-Google
Disallow: /*.aspx
Disallow: /_controls/
Disallow: /_master/
Disallow: /Account/
Disallow: /Admin/
Disallow: /Error/
Disallow: /Login/
Disallow: /Public/

Browser errors
106253-errorparte1.png


106209-errorparte2.png




Thank you very much for your help

dotnet-aspnet-general
errorparte1.png (54.3 KiB)
errorparte2.png (61.3 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

BruceBarker-8516 avatar image
0 Votes"
BruceBarker-8516 answered DiegoLpez-2745 commented

Your site does not allow anonymous access to robots.txt, so it redirecting to the login page which is not a valid robots.txt format. As SEO is only about anonymous access pages, once you fix robots.txt to allow anonymous access, it should only list pages that also allow anonymous access.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I understand, thank you very much. Now if you want to modify the robots.txt file for anonymous access how can I do it?

0 Votes 0 ·
Bruce-SqlWork avatar image
0 Votes"
Bruce-SqlWork answered

That would depend on what you are using for authentication and how it is configured. If (as it appears) your site requires authentication to access, then you don't need to worry about SEO, because the site will not be crawled, as the crawler can not login.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.