Integrate ngrok into ASP.NET Core startup and automatically update your webhook URLs

Niels Swimberghe - 2/15/2022 - .NET

This blog post was written for Twilio and originally published at the Twilio blog.

When you are developing web applications on your local machine, you sometimes need your application to be reachable from the internet. One of the most common reasons to do this is to develop webhooks.

Webhooks are a way to be notified by an external service when an event has occurred. Instead of you sending an HTTP request to that service, the service sends an HTTP request to your public web service.

To develop webhooks locally, you can use a tunnel service like ngrok which creates a tunnel between your local network and the internet. However, when you're using ngrok's free plan, ngrok will create a random, public URL anytime you restart your tunnel. This means that you need to update your webhooks with the new URL anytime it changes. If updating your webhook URLs takes a bunch of clicks and keystrokes, this can be quite a hassle.

Luckily, you can avoid the repetitive work by automating this! In this tutorial, you'll learn how to automatically start ngrok when your ASP.NET Core application starts. Then, you'll learn how to grab the random ngrok URL and use the URL to configure Twilio's webhooks automatically.

Prerequisites #

You will need these items to follow along:

An OS that supports .NET (Windows/macOS/Linux)
.NET 6 SDK
A code editor or IDE (Recommended: VS Code with the C# plugin, Visual Studio, or JetBrains Rider)
The Ngrok CLI
A free Ngrok account (optional)
A free Twilio account (If you register here, you'll receive $10 in Twilio credit when you upgrade to a paid account!)

You can find the source code for this tutorial on GitHub. Use the source code if you run into any issues, or submit an issue on this GitHub repo if you run into problems.

Create an ASP.NET Core web project #

Open your preferred shell, and use the commands below to create a new folder named NgrokAspNet, and navigate to it:

mkdir NgrokAspNet
cd NgrokAspNet

Use the .NET CLI to create a new empty web project:

dotnet new web

Your new project contains one C# file, named Program.cs:

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

The program creates a new web application with a single endpoint responding with "Hello World". Head back to your shell and start the project using the .NET CLI:

dotnet run

The output should look like this:

Building...
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: https://localhost:7121
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:5033
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/nswimberghe/NgrokAspNet/

Take note of the two localhost URLs chosen for you, when you generated the web project.
Pick one of the URLs and open it in a web browser. You should see "Hello World!" displayed in your browser.

Leave the .NET project running and open a new shell to run the following commands!

Use ngrok to tunnel your local web server to the internet #

You can run the ngrok CLI tool on your machine to tunnel the local URL to a public URL that looks similar to https://66a605a7ced5.ngrok.io, with a different subdomain. Every time you start a new tunnel using ngrok, the subdomain is different.

In your new shell, run the following command:

ngrok http [YOUR_HTTP_SERVER_URL]

Replace [YOUR_HTTP_SERVER_URL] with the web server URL starting with http://localhost.

This will start a new tunnel to a new public URL that you can find in the displayed output:

Ngrok output showing the two public URLs forwarding the local web server URLs

Switch back to the web browser and navigate to one of the Forwarding URLs listed in your shell. Some browsers may warn you that this is a deceptive site, which you can disregard in this case. The browser should once again return "Hello World", but this time via the public URL. This means you can share this URL with anyone, and they will also be able to communicate with your local web server. This also means webhooks can reach your local server.

You can also tunnel HTTPS URLs, but you need to sign up for an ngrok account (free) and authenticate your ngrok CLI tool. Once you have done that, you can also run the ngrok http command with the URL starting with https://localhost instead.

In addition to the forwarding URLs, there's also a Web Interface URL displayed. This is where you can access ngrok's local dashboard and API. Switch back to your browser and navigate to http://localhost:4040. Here you will find the forwarding URLs once again, and also a log of all the HTTP requests coming through the tunnel.

Stop the ngrok process by pressing ctrl + c and close this shell instance. Switch to your other shell and stop the .NET project by pressing ctrl + c.

Start ngrok automatically during ASP.NET Core startup #

You were able to publicly serve your web application by running your ASP.NET project and then running ngrok in a separate shell. This workflow can be optimized by integrating ngrok into the startup process of your ASP.NET project.

To start the ngrok tunnel programmatically, you'll need to run the ngrok CLI command from code. You could use the Process .NET APIs, but there's an open-source library that makes interacting with CLI tools and processes easier: CliWrap.

Add the CliWrap NuGet package using the .NET CLI:

dotnet add package CliWrap

The CliWrap NuGet package is at version 3.4.0 at the time of writing this.

Create a new C# file in the NgrokAspNet project directory, named TunnelService.cs, and add the following code:

using System.Text.Json.Nodes;
using CliWrap;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;

namespace NgrokAspNet;

public class TunnelService : BackgroundService
{
    private readonly IServer server;
    private readonly IHostApplicationLifetime hostApplicationLifetime;
    private readonly IConfiguration config;
    private readonly ILogger<TunnelService> logger;

    public TunnelService(
        IServer server,
        IHostApplicationLifetime hostApplicationLifetime,
        IConfiguration config,
        ILogger<TunnelService> logger
    )
    {
        this.server = server;
        this.hostApplicationLifetime = hostApplicationLifetime;
        this.config = config;
        this.logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await WaitForApplicationStarted();

        var urls = server.Features.Get<IServerAddressesFeature>()!.Addresses;
        // Use https:// if you authenticated ngrok, otherwise, you can only use http://
        var localUrl = urls.Single(u => u.StartsWith("http://"));

        logger.LogInformation("Starting ngrok tunnel for {LocalUrl}", localUrl);
        var ngrokTask = StartNgrokTunnel(localUrl, stoppingToken);

        var publicUrl = await GetNgrokPublicUrl();
        logger.LogInformation("Public ngrok URL: {NgrokPublicUrl}", publicUrl);

        await ngrokTask;
        
        logger.LogInformation("Ngrok tunnel stopped");
    }

    private Task WaitForApplicationStarted()
    {
        var completionSource = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
        hostApplicationLifetime.ApplicationStarted.Register(() => completionSource.TrySetResult());
        return completionSource.Task;
    }

    private CommandTask<CommandResult> StartNgrokTunnel(string localUrl, CancellationToken stoppingToken)
    {
        var ngrokTask = Cli.Wrap("ngrok")
            .WithArguments(args => args
                .Add("http")
                .Add(localUrl)
                .Add("--log")
                .Add("stdout"))
            .WithStandardOutputPipe(PipeTarget.ToDelegate(s => logger.LogDebug(s)))
            .WithStandardErrorPipe(PipeTarget.ToDelegate(s => logger.LogError(s)))
            .ExecuteAsync(stoppingToken);
        return ngrokTask;
    }

    private async Task<string> GetNgrokPublicUrl()
    {
        using var httpClient = new HttpClient();
        for (var ngrokRetryCount = 0; ngrokRetryCount < 10; ngrokRetryCount++)
        {
            logger.LogDebug("Get ngrok tunnels attempt: {RetryCount}", ngrokRetryCount + 1);

            try
            {
                var json = await httpClient.GetFromJsonAsync<JsonNode>("http://127.0.0.1:4040/api/tunnels");
                var publicUrl = json["tunnels"].AsArray()
                    .Select(e => e["public_url"].GetValue<string>())
                    .SingleOrDefault(u => u.StartsWith("https://"));
                if (!string.IsNullOrEmpty(publicUrl)) return publicUrl;
            }
            catch
            {
                // ignored
            }

            await Task.Delay(200);
        }

        throw new Exception("Ngrok dashboard did not start in 10 tries");
    }
}

The TunnelService class will be responsible for starting a tunnel using the ngrok CLI, and later on it will also configure Twilio webhooks.

This is a lot of code, so let's dissect it piece by piece.

    public TunnelService(
        IServer server,
        IHostApplicationLifetime hostApplicationLifetime,
        IConfiguration config,
        ILogger<TunnelService> logger
    )
    {
        this.server = server;
        this.hostApplicationLifetime = hostApplicationLifetime;
        this.config = config;
        this.logger = logger;
    }

The constructor accepts multiple parameters that will be provided by the dependency injection container built into ASP.NET Core. All the parameters are stored in private fields, so they are accessible throughout the class.

The server parameter contains information about the web server currently being started. Once the web server is started, you can retrieve the local URLs from the server field.
The hostApplicationLifetime parameter lets you hook into the different lifecycle events (started/stopping/stopped).
The config parameter will contain all the configuration passed into the .NET application through command-line arguments, environment variables, JSON files, user-secrets, etc. The config isn't used right now, but it will be used in an upcoming section.
The logger parameter will be used to log any information relevant to running the tunnel.

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await WaitForApplicationStarted();

        var urls = server.Features.Get<IServerAddressesFeature>()!.Addresses;
        // Use https:// if you authenticated ngrok, otherwise, you can only use http://
        var localUrl = urls.Single(u => u.StartsWith("http://"));

        logger.LogInformation("Starting ngrok tunnel for {LocalUrl}", localUrl);
        var ngrokTask = StartNgrokTunnel(localUrl, stoppingToken);

        var publicUrl = await GetNgrokPublicUrl();
        logger.LogInformation("Public ngrok URL: {NgrokPublicUrl}", publicUrl);

        await ngrokTask;
        
        logger.LogInformation("Ngrok tunnel stopped");
    }

TunnelService inherits from the abstract class BackgroundService which is why you need to implement the abstract method ExecuteAsync. ExecuteAsync is the main method of this class and will be invoked as the web application is starting. ExecuteAsync will wait for the web application to have started using WaitForApplicationStarted, and then grab the local URLs. A single URL will be taken from the local URLs. If you authenticated ngrok earlier, you can use the HTTPS URL instead of the HTTP URL by replacing "http://" with "https://".

Next, the ngrok tunnel will be started, then the public ngrok URL will be retrieved, and finally the Task for running the ngrok CLI is awaited. When the web application stops, the ngrok process will also be stopped, which will complete the ngrokTask.

    private Task WaitForApplicationStarted()
    {
        var completionSource = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
        hostApplicationLifetime.ApplicationStarted.Register(() => completionSource.TrySetResult());
        return completionSource.Task;
    }

WaitForApplicationStarted will create an awaitable Task that will be completed when the ApplicationStarted event is triggered. Oddly, the lifecycle events on IHostApplicationLifetime are not using delegates or C# events, but instead they are CancellationToken's.

You can pass in a lambda or delegate to the CancellationToken.Register method which will be invoked when the CancellationToken is canceled. In case of the cancellation token stored in the IHostApplicationLifetime.ApplicationStarted property, when the token is canceled, this means the application has started. How (un)intuitive, am I right?

To make this more intuitive to use, you can create a TaskCompletionSource and set its result in the hostApplicationLifetime.ApplicationStarted.Register callback. This will set the Task as completed, which in this case will be when the application has started.

If all of this is a little confusing, the important thing to take away is that await WaitForApplicationStarted() will wait for the web application to have started.

    private CommandTask<CommandResult> StartNgrokTunnel(string localUrl, CancellationToken stoppingToken)
    {
        var ngrokTask = Cli.Wrap("ngrok")
            .WithArguments(args => args
                .Add("http")
                .Add(localUrl)
                .Add("--log")
                .Add("stdout"))
            .WithStandardOutputPipe(PipeTarget.ToDelegate(s => logger.LogDebug(s)))
            .WithStandardErrorPipe(PipeTarget.ToDelegate(s => logger.LogError(s)))
            .ExecuteAsync(stoppingToken);
        return ngrokTask;
    }

StartNgrokTunnel will use the CliWrap library to run the ngrok CLI. The resulting command will look like this:

ngrok http [YOUR_LOCAL_SERVER_URL] --log stdout

This command will start the ngrok tunnel like before, but with the addition of the --log stdout argument. This log argument instructs ngrok to log to standard output which can then be captured through WithStandardOutputPipe. The standard output and error output will be piped to the logger.

A critical detail is that the stoppingToken is passed to ExecuteAsync. The stoppingToken will be canceled when the application is being stopped. You can use this token to gracefully handle when the application is about to be shutdown. By passing the stoppingToken to ExecuteAsync, the ngrok process will also be stopped when the application is stopped. Thus, you won't have any ngrok child processes lingering around.

    private async Task<string> GetNgrokPublicUrl()
    {
        using var httpClient = new HttpClient();
        for (var ngrokRetryCount = 0; ngrokRetryCount < 10; ngrokRetryCount++)
        {
            logger.LogDebug("Get ngrok tunnels attempt: {RetryCount}", ngrokRetryCount + 1);

            try
            {
                var json = await httpClient.GetFromJsonAsync<JsonNode>("http://127.0.0.1:4040/api/tunnels");
                var publicUrl = json["tunnels"].AsArray()
                    .Select(e => e["public_url"].GetValue<string>())
                    .SingleOrDefault(u => u.StartsWith("https://"));
                if (!string.IsNullOrEmpty(publicUrl)) return publicUrl;
            }
            catch
            {
                // ignored
            }

            await Task.Delay(200);
        }

        throw new Exception("Ngrok dashboard did not start in 10 tries");
    }

The GetNgrokPublicUrl will fetch the public HTTPS URL and return it. You can get the public tunnel URLs by requesting it from the local ngrok API at http://127.0.0.1:4040/api/tunnels.

Unfortunately, when the ngrok CLI is started, that doesn't mean the tunnel is ready yet. That's why this code is surrounded in a loop that will try to get the public URL up to 10 times, every 200 milliseconds. Feel free to change the 200ms delay and the retryCount to whatever suits your needs.

The TunnelService class is complete, but you still need to configure the web application to run it in the background. Update Program.cs, based on the highlighted lines below:

var builder = WebApplication.CreateBuilder(args);
if (builder.Environment.IsDevelopment()) 
    builder.Services.AddHostedService<NgrokAspNet.TunnelService>();
var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

The TunnelService will be configured to run in the background, but only when the application is run in a development environment. After all, you only want to use the ngrok tunnel for local development.

It doesn't make sense to run this in staging or production, and would probably cause trouble if it did run. Instead of only running this is in a development environment, you could also add a more explicit configuration element, but that's up to you!

That's it! Run the application using the .NET CLI and watch the output:

dotnet run

The public ngrok URL will be logged to the output like this: "Public ngrok URL: https://6797-72-66-29-154.ngrok.io". Grab the public ngrok URL and navigate to it in the browser. You will see, once again, "Hello World!".

Update Twilio Webhooks automatically with ngrok URLs #

Get started with Twilio #

If you haven't already, you'll need to set up the following with Twilio:

Go and buy a new phone number from Twilio. The cost of the phone number will be applied to your free promotional credit.Make sure to take note of your new Twilio phone number. You'll need it later on!
If you are using a trial Twilio account, you can only send text messages to Verified Caller IDs. Verify your phone number or the phone number you want to SMS if it isn't on the list of Verified Caller IDs.
Lastly, you'll need to find your Twilio Account SID and Auth Token. Navigate to your Twilio account page and take note of your Twilio Account SID and Auth Token located at the bottom left of the page.

Update Twilio Phone Number Webhooks #

The Twilio SDK for C# and .NET will help you interact with Twilio's APIs and respond to webhooks. Add the Twilio NuGet package to your project:

dotnet add package twilio

Go back to your code editor and open the TunnelService.cs file. Update the using statements at the top of the file, to include these three new Twilio references:

using System.Text.Json.Nodes;
using CliWrap;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Twilio.Clients;
using Twilio.Rest.Api.V2010.Account;
using Twilio.Types;

Update the ExecuteAsync method to invoke the asynchronous ConfigureTwilioWebhook method after logging the public ngrok URL:

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await WaitForApplicationStarted();

        var urls = server.Features.Get<IServerAddressesFeature>()!.Addresses;
        // Use https:// if you authenticated ngrok, otherwise, you can only use http://
        var localUrl = urls.Single(u => u.StartsWith("http://"));

        logger.LogInformation("Starting ngrok tunnel for {LocalUrl}", localUrl);
        var ngrokTask = StartNgrokTunnel(localUrl, stoppingToken);

        var publicUrl = await GetNgrokPublicUrl();
        logger.LogInformation("Public ngrok URL: {NgrokPublicUrl}", publicUrl);

        await ConfigureTwilioWebhook(publicUrl);
        
        await ngrokTask;
        
        logger.LogInformation("Ngrok tunnel stopped");
    }

Add the asynchronous ConfigureTwilioWebhook method after the GetNgrokPublicUrl method:

    private async Task ConfigureTwilioWebhook(string publicUrl)
    {
        var twilioClient = new TwilioRestClient(config["TwilioAccountSid"], config["TwilioAuthToken"]);
        var phoneNumber = (await IncomingPhoneNumberResource.ReadAsync(
            phoneNumber: new PhoneNumber(config["TwilioPhoneNumber"]),
            limit: 1,
            client: twilioClient
        )).Single();
        phoneNumber = await IncomingPhoneNumberResource.UpdateAsync(
            phoneNumber.Sid,
            voiceUrl: new Uri($"{publicUrl}/voice"), voiceMethod: Twilio.Http.HttpMethod.Post,
            smsUrl: new Uri($"{publicUrl}/message"), smsMethod: Twilio.Http.HttpMethod.Post,
            client: twilioClient
        );
        logger.LogInformation(
            "Twilio Phone Number {TwilioPhoneNumber} Voice URL updated to {TwilioVoiceUrl}",
            phoneNumber.PhoneNumber,
            phoneNumber.VoiceUrl
        );
        logger.LogInformation(
            "Twilio Phone Number {TwilioPhoneNumber} Message URL updated to {TwilioMessageUrl}",
            phoneNumber.PhoneNumber,
            phoneNumber.SmsUrl
        );
    }

The ConfigureTwilioWebhook method receives the public ngrok URL as a parameter. The Account SID and Auth Token are retrieved from the config field and passed into the constructor of TwilioRestClient.

You can use API keys to authenticate instead of using the Account SID and the Auth Token. API keys have fewer permissions and can be revoked more easily, which makes them a safer option.

The Twilio phone number details are requested using TwilioRestClient, and then the phone number details are used to update the voice webhook URL and the SMS webhook URL. The voice and SMS webhook URLs will be set to the public tunnel URL with /voice and /message appended to it, respectively.

The project now depends on the TwilioAccountSid, TwilioAuthToken, and TwilioPhoneNumber configuration element, but they haven't been configured yet. You can use .NET user secrets to configure these types of sensitive configuration.

Initialize user secrets for your project using the .NET CLI:

dotnet user-secrets init

Run the following command to configure the secrets:

dotnet user-secrets set TwilioAccountSid [YOUR ACCOUNT SID]
dotnet user-secrets set TwilioAuthToken [YOUR AUTH TOKEN]
dotnet user-secrets set TwilioPhoneNumber [YOUR TWILIO PHONE NUMBER]

Replace [YOUR ACCOUNT SID] with your Twilio Account SID, [YOUR AUTH TOKEN] with your Twilio Auth Token, and [YOUR TWILIO PHONE NUMBER] with your Twilio Phone Number.

Test out your work so far by running the application:

dotnet run

You should see additional output that looks like this: "Twilio Phone Number +1234567890 Voice URL updated to https://5fb3-72-66-29-154.ngrok.io/voice" and "Twilio Phone Number +1234567890 Message URL updated to https://5fb3-72-66-29-154.ngrok.io/message"

Respond to Twilio webhooks #

Once the webhook URLs are set, Twilio will send HTTP requests to your public URLs whenever a phone call or SMS goes to your Twilio phone number.
You need to accept these HTTP requests for /voice and /message, and then respond with TwiML instructions. Update Program.cs based on the highlighted lines in the code below:

using Twilio.TwiML;

var builder = WebApplication.CreateBuilder(args);
if (builder.Environment.IsDevelopment()) 
    builder.Services.AddHostedService<NgrokAspNet.TunnelService>();
var app = builder.Build();

app.MapGet("/", () => "Hello World!");
app.MapPost("/voice", () =>
{
    var response = new VoiceResponse();
    response.Say("Hello World!");
    return Results.Text(response.ToString(), "application/xml");
});
app.MapPost("/message", () =>
{
    var response = new MessagingResponse();
    response.Message("Hello World!");
    return Results.Text(response.ToString(), "application/xml");
});

app.Run();

When Twilio sends an HTTP POST request to /voice, the endpoint will respond with the following TwiML:

<?xml version="1.0" encoding="utf-8"?>
<Response>
  <Say>Hello World!</Say>
</Response>

As a result, Twilio will transcribe "Hello World!" to audio and stream it to the caller.

When Twilio sends an HTTP POST request to /message, the endpoint will respond with the following TwiML:

<?xml version="1.0" encoding="utf-8"?>
<Response>
  <Message>Hello World!</Message>
</Response>

As a result, Twilio will respond with a text message saying "Hello World!".

Testing the Twilio webhooks #

If everything went well, you are now able to develop and test webhooks by running a single command dotnet run. Start the application using the .NET CLI:

dotnet run

Wait for the webhook URLs to be updated, and then call and/or text your Twilio Phone Number.
If you call, you should hear "Hello World!", and if you text, you should receive a text message saying "Hello World!".

How to integrate ngrok into ASP.NET and automatically update your webhooks #

In this tutorial, you learned how to streamline your webhook development process by integrating ngrok into your ASP.NET Core startup and automatically updating your webhooks, using these steps:

Get your local ASP.NET URLs
Use a BackgroundService to run the ngrok tunnel
Fetch the ngrok forwarding URL from ngrok's local API
Update your webhooks URLs using the ngrok forwarding URL

Twilio has a lot of other products you can integrate into your applications. Check out this tutorial on how to make phone calls from Blazor WebAssembly with Twilio Voice and Twilio Client.

Additional resources #

Check out the following resources for more information on the topics and tools presented in this tutorial:

TwiML for Programmable Voice – Learn more about TwiML and how you can use TwiML to handle phone calls.

TwiML for Programmable SMS – Learn more about TwiML and how you can use TwiML to respond to text messages.

Source Code for this tutorial on GitHub - Use this source code if you run into any issues, or submit an issue on this GitHub repo if you run into problems.

Niels Swimberghe is a Belgian American software engineer and technical content creator at Twilio. Get in touch with Niels on Twitter @RealSwimburger and follow Niels’ personal blog on .NET, Azure, and web development at swimburger.net.