--- title: "Integrate IndexNow with Umbraco CMS to submit content to search engines" description: "Umbraco is very extensible for ASP.NET developers. Learn how to integrate IndexNow into the Umbraco backoffice, so you can easily submit Umbraco content URLs to search engines that they will (re)crawl." canonical: "https://swimburger.net/blog/umbraco/integrate-indexnow-with-umbraco-cms-to-submit-content-to-search-engines" published: "2022-02-07T00:00:00.000Z" updated: "2022-02-07T22:25:19.560Z" --- # Integrate IndexNow with Umbraco CMS to submit content to search engines One of Umbraco's superpowers has always been how easy it is to extend the CMS as a .NET web developer. As an ASP.NET developer, you can develop your websites on top of Umbraco's content database, but you can also extend the CMS itself and its [backoffice](https://our.umbraco.com/documentation/Fundamentals/Backoffice/). The backoffice is Umbraco's web interface for managing content, developing the website, and other administrative tasks. You can empower the backoffice users by adding more functionality, making it the central hub for managing your website. For example, I added Google Analytics to the backoffice, capabilities to clear Cloudflare's cache for specific Umbraco content, and [just recently, I added IndexNow integration](https://twitter.com/RealSwimburger/status/1487910084889960454) to easily submit content to search engines for crawling.  In this tutorial, you'll learn how to integrate IndexNow, so you can easily submit Umbraco content URLs to search engines that they will (re)crawl. Along the way you'll also learn how to: 1. hook into Umbraco's startup 2. extend the backoffice menu's 3. add JavaScript to the backoffice 4. add an API controller for authenticated backoffice users 5. submit an Umbraco content URL to IndexNow The result of the IndexNow integration will look like this:  > I spent this afternoon adding IndexNow support for Bing and Yandex to my Umbraco website. > Will write a blog post about it ... someday 🙈[#dotnet](https://twitter.com/hashtag/dotnet?src=hash&ref_src=twsrc%5Etfw) [#aspnet](https://twitter.com/hashtag/aspnet?src=hash&ref_src=twsrc%5Etfw) [#umbraco](https://twitter.com/hashtag/umbraco?src=hash&ref_src=twsrc%5Etfw) [pic.twitter.com/GyxxB03Ewi](https://t.co/GyxxB03Ewi) > > — Niels Swimburger.NET 🍔 (@RealSwimburger) [January 30, 2022](https://twitter.com/RealSwimburger/status/1487910084889960454?ref_src=twsrc%5Etfw) ## What's IndexNow?  [IndexNow](https://www.indexnow.org/index) allows software developers to work with a single simple protocol, for all search engines who adopt it, to requests search engines to crawl your website's URLs. What's more, when you submit URLs to one search engine, that search engine will inform all other search engines with IndexNow support. That means you only have to make an API call to one IndexNow service, not to every search engine's IndexNow service. ### How to implement IndexNow  1. Create a **key** between 8 and 128 hexadecimal characters. Allowed characters: lowercase characters (a-z), uppercase characters (A-Z), numbers (0-9), and dashes (-). You can choose this key yourself, or you can [generate a valid key on Bing's IndexNow site](https://www.bing.com/indexnow#generateApiKey) for your convenience. 2. Upload a text file to the root of your website. The name of the file is **your key with the "*.txt*" extension**, and the **contents of the file is also your key**. 3. Submit HTTP requests to any IndexNow service: - IndexNow.org: [https://api.indexnow.org/indexnow](https://api.indexnow.org/indexnow) - Bing IndexNow: [https://www.bing.com/indexnow](https://www.bing.com/indexnow) - Yandex IndexNow: [https://yandex.com/indexnow](https://yandex.com/indexnow) You can send HTTP requests 2 ways. The first and simplest way is to send a GET HTTP request with two querystring parameters: - `url`: this is the URL you want search engines to crawl - `key`: this is the key you generated earlier This will submit a single URL to IndexNow. To send multiple URLs, you can use the second way. The second way is to send a POST HTTP request with the following JSON body: ```json { "host": "example.org", "key": "db132248f7cb4d309ee9d2aa54131af6", "keyLocation": "https://example.org/db132248f7cb4d309ee9d2aa54131af6.txt", "urlList": [ "https://example.org/url1", "https://example.org/folder/url2", "https://example.org/url3" ] } ``` Here's what the JSON properties should be: - `host`: The domain name of your website. The `host` must match the URLs you are submitting. - `key`: The IndexNow key you generated. - `keyLocation`: Where you host your key text file. This property is optional if you store the key text file at the root of your website. - `urlList`: An array of URLs you want to submit to search engines for crawling. Read [this post to learn more about IndexNow and why you should care as a web developer, content editor, marketeer, or webmaster](https://swimburger.net/blog/web/an-introduction-to-indexnow-and-why-you-should-care). ## Prerequisites  To follow along, you'll need the following things: - Windows OS - .NET Framework - A .NET framework friendly IDE (Visual Studio, JetBrains Rider, etc.) - An Umbraco 8 website (this tutorial used the [NuGet installation method](https://our.umbraco.com/Documentation/Fundamentals/Setup/Install/install-umbraco-with-nuget) with the default starter kit) You can find the [source code for the IndexNow integration on this GitHub repository](https://github.com/Swimburger/UmbracoIndexNow). If you run into any problems, feel free to submit an issue or reach out through social media. This tutorial uses Umbraco 8 which only supports Windows using the .NET Framework, but what you'll learn here you can also apply to Umbraco 9 which use .NET (Core) and supports Windows, macOS, and Linux. ## Hooking into Umbraco's startup  You'll need to be able to run code at Umbraco's startup to extend the backoffice menu.  Open your Umbraco website using an editor or IDE and create a new folder called *IndexNow*. In this folder, add a new C# file called *IndexNowComposer.cs* with the following contents: ```csharp using Umbraco.Core; using Umbraco.Core.Composing; namespace UmbracoIndexNow.IndexNow { public class IndexNowComposer : IUserComposer { public void Compose(Composition composition) => composition.Components().Append(); } } ``` During startup, Umbraco will scan for all composers and run them. **Composers** will add **components** and thus running all composers will create a **composition of components**. The composition of components is how Umbraco configures itself, and where you can add new components or replace existing components of Umbraco. The `IndexNowComposer` will append the `IndexNowComponent` to the list of existing components. The `IndexNowComponent` does not exist yet, so create a new file at *IndexNow/IndexNowComponent.cs* with the following content: ```csharp using Umbraco.Core.Composing; namespace UmbracoIndexNow.IndexNow { public class IndexNowComponent : IComponent { public void Initialize() { } public void Terminate() { } } } ``` In the component, there are two methods: `Initialize` and `Terminate`. The `Initialize` method executes during Umbraco's startup which is when you can run code to extend Umbraco. The `Terminate` method runs when the application is stopped, and this will allow you to cleanup whatever resources you created during initialization. ## Extending the Umbraco backoffice menu  To extend the backoffice menu, you actually need to do something in the `IndexNowComponent`, so update the *IndexNow/IndexNowComponent.cs* file as below: ```csharp using Umbraco.Core.Composing; using Umbraco.Web.Models.Trees; using Umbraco.Web.Trees; namespace UmbracoIndexNow.IndexNow { public class IndexNowComponent : IComponent { public void Initialize() => TreeControllerBase.MenuRendering += OnMenuRendering; public void Terminate() => TreeControllerBase.MenuRendering -= OnMenuRendering; private void OnMenuRendering(TreeControllerBase sender, MenuRenderingEventArgs e) { if (sender.TreeAlias != "content") { return; } var content = sender.Umbraco.Content(int.Parse(e.NodeId)); if (content == null) { return; } var contentType = content.ContentType; var compositions = contentType.CompositionAliases; if (compositions.Contains("contentBase") || compositions.Contains("navigationBase") || contentType.Alias == "home") { var i = new MenuItem("submitToIndexNow", "Submit To IndexNow"); i.ExecuteJsMethod("submitToIndexNow(node)"); i.Icon = "share-alt"; e.Menu.Items.Add(i); } } } } ``` Now the `IndexNowComponent` registers a callback on the `TreeControllerBase.MenuRendering` event during initialization and unregisters the callback during termination. Every time a user opens the menu for a tree in the backoffice, the `OnMenuRendering` method will be called. For example, when you right-click on a content node in the content tree. Though, this method will also be called for other trees in the backoffice such as settings, members, etc. Since you're only interested in changing the menu in the content tree, the code above checks if this is the 'content' tree and immediately backs out if it is not. Now that you know you're in the content tree, you know that the `NodeId` will point to Umbraco content. Thus, you can fetch the Umbraco content by the `NodeId`. Although not all content nodes in the tree will be published. If the content isn't published, the `content` variable will be null. If so, you can safely back out as it doesn't make sense to submit unpublished content to IndexNow. Which leads you to the question, what type of content do you want to submit to IndexNow? That depends on how you built your content architecture. For the starter kit, only content that either is composed of the `contentBase` or `navigationBase` composisition makes sense to submit to IndexNow. The only exception being the home page, which doesn't have compositions for some reason. Now that you've verified the selected content makes sense to submit to IndexNow, you can finally add a menu item to the menu. You can configure the menu item to do different things like open a dialog, navigate to a route, or execute a JavaScript function. As a lazy developer, the easiest option seemed to execute a JavaScript function. When this menu item is clicked, a global JavaScript function will be invoked named `submitToIndexNow`. If you want to do something else with menus and tree, check out [the Umbraco docs for extending trees](https://our.umbraco.com/Documentation/Extending/Section-Trees/trees-v8). ## Add JavaScript to submit Umbraco Node to web API  Interestingly, Umbraco will use the [eval-function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval) to evaluate the JavaScript string passed to `ExecuteJsMethod` and run it. This also means you have access to whatever variables are within scope when Umbraco runs your JavaScript expression. Most importantly, there is a `node` variable which holds the details about the selected node in the tree. That's why the `node` variable is passed as an argument to the `submitToIndexNow` function. Create a new folder named *IndexNow* under *App\_Plugins,* and then create a new JavaScript file *IndexNow.js* with the following contents: ```javascript function submitToIndexNow(node) { fetch(`/umbraco/backoffice/api/IndexNow/Submit?nodeId=${node.id}`, { method: 'post', }) .then(response => response.json()) .then(response => { if (response.success === true) { UmbSpeechBubble.ShowMessage('success', 'IndexNow', 'Submitted successfully'); } else { console.log(response.errorMessage); UmbSpeechBubble.ShowMessage('error', 'IndexNow', response.errorMessage); } }) .catch(err => { console.log(err); UmbSpeechBubble.ShowMessage('error', 'IndexNow', 'Submit failed'); }); } ``` The `submitToIndexNow` sends an HTTP request to */umbraco/backoffice/api/IndexNow/Submit* passing in the `id` of the selected tree node. The application currently doesn't listen to this endpoint, but later on the HTTP request will be handled by the `Submit` method on the `IndexNowController`. The response from this controller will be a JSON object with two properties: `success` and `errorMessage`. `success` is a boolean indicating whether the content was successfully submitted to IndexNow, and `errorMessage` will provide a user-friendly message if `success` is false. In both cases, a little toast will be displayed to the user by calling the `UmbSpeechBubble.ShowMessage` function provided by Umbraco. ![Green highlighted message saying "IndexNow: Submitted successfully"](https://swimburger.net/cdn-cgi/image/format=auto,quality=90,width=391/media/is2fqln1/toast.png) In case of an error, the error is logged to the console and a toast will also be shown with a generic failure message. For Umbraco to actually load this JavaScript, you need to create a plugin manifest. Create a new file name *package.manifest* with the contents below: ```json { "javascript": [ "/App_Plugins/IndexNow/IndexNow.js" ] } ``` With these two files, Umbraco will start including your JavaScript function in the backoffice.  ## Add backoffice API controller to submit URL to IndexNow  You'll need to create a controller to handle the HTTP request made from the `submitToIndexNow` function. Create a new file named *IndexNowController.cs* in the *IndexNow* folder (not *App\_Plugins/IndexNow*) with the following contents: ```csharp using Newtonsoft.Json; using System.Net.Http; using System.Text; using System.Threading.Tasks; using System.Web.Http; using Umbraco.Core.Logging; using Umbraco.Core.Models.PublishedContent; using Umbraco.Web; using Umbraco.Web.WebApi; namespace UmbracoIndexNow.IndexNow { public class IndexNowController : UmbracoAuthorizedApiController { public static readonly HttpClient httpClient = new HttpClient(); private readonly ILogger logger; public IndexNowController(ILogger logger) { this.logger = logger; } [HttpPost] public async Task Submit(int nodeId) { // get URL of the Umbraco content var nodeUrl = Umbraco.Content(nodeId).Url(mode: UrlMode.Relative); // create IndexNow object var indexNowObject = new { host = IndexNowConfiguration.Host, key = IndexNowConfiguration.Key, keyLocation = IndexNowConfiguration.KeyLocation, urlList = new string[] { $"{IndexNowConfiguration.BaseUrl}{nodeUrl}" } }; // convert IndexNow object to JSON string var jsonString = JsonConvert.SerializeObject(indexNowObject); // send JSON string to IndexNow service var content = new StringContent(jsonString, Encoding.UTF8, "application/json"); var response = await httpClient.PostAsync(IndexNowConfiguration.Url, content); // if response is status code 200, return object with `success` = true if (response.IsSuccessStatusCode) { return new { success = true }; } // if response statuscode is not 200 // log an error message logger.Error( "Submitting URL to {IndexNowUrl} failed, status code: {StatusCode}", IndexNowConfiguration.Url, response.StatusCode ); // log the response body as error for debugging purposes var responseContent = await response.Content.ReadAsStringAsync(); logger.Error("IndexNow response: {IndexNowResponse}", responseContent); // return object with `success` = false and a user-friendly error message return new { success = false, errorMessage = $"Submitting URL to {IndexNowConfiguration.Url} failed, status code: {response.StatusCode}" }; } } } ``` By inheriting from `UmbracoAuthorizedController`, this controller can only be used by authenticated backoffice users and the default routing will be `/umbraco/backoffice/api/[controller]/[action]` which will resolve to */umbraco/backoffice/api/IndexNow/Submit* in this case. The `Submit` action receives the `nodeId` which was passed as a querystring parameter and uses it to fetch the relative URL for the Umbraco content. The relative URL is combined with a base URL to create an absolute URL. The base URL comes from the `IndexNowConfiguration` class which will be introduced next.  Next, an anonymous object is created following the IndexNow JSON schema using the absolute URL and more properties coming from the `IndexNowConfiguration` class. The object is converted to a JSON string and submitted using an HTTP POST request to the IndexNow service. If the response returns an HTTP status code 200, a JSON object `{"success" = true}` is returned. Otherwise, an error message is logged, the response body is logged, and the action will return `{"success" = false, "errorMessage" = "..."}`. The last missing piece is the `IndexNowConfiguration` class. Create a new file `IndexNowConfiguration.cs` in the *IndexNow* folder (not *App\_Plugins/IndexNow*) with the following contents: ```csharp using System.Configuration; namespace UmbracoIndexNow.IndexNow { public static class IndexNowConfiguration { public static string BaseUrl => ConfigurationManager.AppSettings["IndexNow.BaseUrl"]; public static string Url => ConfigurationManager.AppSettings["IndexNow.Url"]; public static string Host => ConfigurationManager.AppSettings["IndexNow.Host"]; public static string Key => ConfigurationManager.AppSettings["IndexNow.Key"]; public static string KeyLocation => ConfigurationManager.AppSettings["IndexNow.KeyLocation"]; } } ``` This static class will resolve all of its static properties using the `ConfigurationManager.AppSettings`. The app settings are populated by the *Web.config* file in your project. Add the configuration to your *Web.config* file like this: ```xml ``` Here's what the properties should be configured as: - `IndexNow.BaseUrl`: this will be used as the base URL to convert your relative URLs to absolute URLs. Configure this with your public website root URL, for example: *[https://swimburger.net](https://swimburger.net)* - `IndexNow.Url`: this is the URL for the IndexNow service you will submit your requests to. You can leave this as is, or swap it out for another IndexNow service. You can find the [other URLs on IndexNow's FAQ](https://www.indexnow.org/faq). - `IndexNow.Key`: this is the key which you need to upload as a text file to the root of your website, and also pass with every HTTP request. You can create a key yourself as explained earlier, or generate a valid key on Bing's IndexNow site for your convenience. - `IndexNow.KeyLocation`: this is the URL pointing to the text file with your key in it, hosted on your public website. There are many other ways to handle configuration, but this is a simple and efficient way. ## Test your IndexNow integration  With everything in place, you can start your Umbraco website and navigate to the backoffice at */umbraco*. Go to the **Content** tab and right-click some of the content. A new menu item should appear saying "Submit to IndexNow". Click on the menu item and the content should be submitted to IndexNow. **Did everything work out as expected? What services have you integrated into Umbraco? I'm curious to find out, let me know!**