A simple editable robots.txt in Episerver

This post will show you how you can easily author a robots.txt content for your Episerver site.

Step 1: Add a CMS property “Robots.txt”

In my case, I added this property to my StartPage

[Display(
   Name = "Robots.txt",
   GroupName = Groups.MetaData,
   Order = 835)]
[UIHint(UIHint.Textarea)]
public virtual string RobotsTxt { get; set; }

Step 2: Create a controller that serves the content of “Robots.txt”

Please note the IContentService below is my own custom interface which returns my StartPage conveniently. You’ll need to replace this with your own implementation of retrieving the content that has the Robots.txt property above.

public class RobotsTxtController : Controller
{
   private readonly IContentService _contentService;

   public RobotsTxtController(IContentService contentService)
   {
      _contentService = contentService;
   }

   [ContentOutputCache]
   public ActionResult Index()
   {
      var startPage = _contentService.GetStartPage();
      string content = startPage.RobotsTxt;
      return Content(content, "text/plain");
   }
}

Step 3: Configure the route mapping for /robots.txt

On your Global.asax.cs file, locate the method RegisterRoutes() and add the following route mapping:

routes.MapRoute(
   "RobotsTxtRoute",
   "robots.txt",
   new { controller = "RobotsTxt", action = "Index" });

And you’re done!

Navigate to your site via your URL/robots.txt