Testing Web Apis with Specflow

Following on from my previous post on using Nullable reference types to aid validation with Web Apis where I mentioned the idea of forming model requirements early on in your design to allow developers to set which properties can be a nullable reference type, I was reminded of the importance of being able to map design requirements to automated tests that will enforce the agreed specification.

TDD with unit tests is a good practice but at such a granular level, it’s not really possible to map them to design requirements, nor are they easily validated by non-devs.

BDD provides a higher level of tests that can fill this gap in which a specification can be mapped into one or more test scenarios and a more complete test can be automated, reflecting a closer to real-life scenario.

Specflow

Specflow is a Cucumber provider for dotnet that allows you to bind business requirements to dotnet code. It effectively allows you to write feature requirements in Gherkin Syntax (Given, When, Then) and automate the execution of those tests through your preferred testing framework (Nunit and XUnit as well as MSTest are all supported).

To get Specflow set up on Visual Studio, you need to install the Specflow extension for your version.

Once you have restarted you can then add a new Specflow feature file form the Add New Item dialog:

You’re given an example to follow in the generated feature file. For our scenario, I’ve gone with an initial test of a missing first name should result in a 400 (Bad request) response. Here’s what the feature text looks like:

Feature: ClientPost
	In order to ensure a valid request body is posted
	As a consumer of this api
	I want to be told the result of my post request

@bodyvalidation
Scenario: Missing First Name
	Given I have the following request body:
	"""
	{
		"lastName": "Smith"
	}
	"""
	When I post this request to the "Client" operation
	Then the result should be a 400 ("Bad Request") response
	And the response body description should include the following text: "The FirstNames field is required"

Unfortunately my WP code plugin doesn’t have a setting for Gherkin so we’re missing the coloured highlights here but let me break it down:

1. Feature – the title plus a description of the intention of the scenarios in the format of “In order to…”, “As a ….”, “I want “
2. Scenario – you can have multiple scenarios and each represents a single test to be run.
3. Given – that’s akin to your arrange (you can use the ‘And’ keyword on a newline if you have more than one thing to setup
4. When – that’s your ‘act’
5. Then (plus subsequent ‘And’s) is where your Assert statements would go

Couple of pointers:

1. Using a triple quote will be treated as a multiline text parameter so good for declaring json request bodies.
2. Adding quotes around a word in any declaration will convert to an input parameter in the generated step definition (see further down)
3. Numbers automatically create parameters in the step definition.

Once you have your first scenario laid out, right click on the editor pane and select ‘Generate Step Definitions’ from the menu:

You’ll be invited to select which statements you want to be converted into step declarations. Leave them all selected and click ‘Generate’. Best to chose a file location relevant to the feature file. Here’s the step definition created from the feature text above:

[Binding]
public class ClientPostSteps
{
	[Given(@"I have the following request body:")]
	public void GivenIHaveTheFollowingRequestBody(string multilineText)
	{
		ScenarioContext.Current.Pending();
	}
	
	[When(@"I post this request to the ""(.*)"" operation")]
	public void WhenIPostThisRequestToTheOperation(string p0)
	{
		ScenarioContext.Current.Pending();
	}
	
	[Then(@"the result should be a (.*) \(""(.*)""\) response")]
	public void ThenTheResultShouldBeAResponse(int p0, string p1)
	{
		ScenarioContext.Current.Pending();
	}
	
	[Then(@"the response body description should include the following text: ""(.*)""")]
	public void ThenTheResponseBodyDescriptionShouldIncludeTheFollowingText(string p0)
	{
		ScenarioContext.Current.Pending();
	}
}

You can see how your feature declarations are mapping to methods with attributes describing the declaration type and any parameters displayed as ‘(.*)’. It’s up to you if you rename your parameters to something more descriptive – it won’t break the tests. My next step is to edit my methods to complete my set up, actions and assertions. Note the ScenarioContext.Current can be used to set or get values through the test, but the example code currently provided also comes with a deprecation warning (!) so best to clean that up as per its suggestion and inject an instance of the current ScenarioContect:

[Binding]
public class ClientPostSteps
{
	private readonly ScenarioContext context;

	public ClientPostSteps(ScenarioContext context)
	{
		this.context = context;
	}

	[Given(@"I have the following request body:")]
	public void GivenIHaveTheFollowingRequestBody(string json)
	{
		// add the request into the scenario context for later use
		context.Set(json, "Request");
	}
	
	[When(@"I post this request to the ""(.*)"" operation")]
	public async Task WhenIPostThisRequestToTheOperation(string operation)
	{
		// retrieve request
		var requestBody = context.Get<string>("Request");

		// set up Http Request Message
		var request = new HttpRequestMessage(HttpMethod.Post, $"/{operation}")
		{
			Content = new StringContent(requestBody)
			{
				Headers = 
				{ 
					ContentType = new MediaTypeHeaderValue("application/json")
				}
			}
		};

		// create an http client
		var client = new HttpClient();

		// let's post
		var response = await client.SendAsync(request).ConfigureAwait(false);

		try
		{
			context.Set(response.StatusCode, "ResponseStatusCode");
			context.Set(response.ReasonPhrase, "ResponseReasonPhrase");
			var responseBody = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
			context.Set(responseBody, "ResponseBody");
		}
		finally
		{
			// move along, move along
		}
	}
	
	[Then(@"the result should be a (.*) \(""(.*)""\) response")]
	public void ThenTheResultShouldBeAResponse(int statusCode, string reasonPhrase)
	{
		Assert.Equal(statusCode, (int)context.Get<HttpStatusCode>("ResponseStatusCode"));
		Assert.Equal(reasonPhrase, context.Get<string>("ResponseReasonPhrase"));
	}
	
	[Then(@"the response body description should include the following text: ""(.*)""")]
	public void ThenTheResponseBodyDescriptionShouldIncludeTheFollowingText(string error)
	{
		Assert.True(context.Get<string>("ResponseBody").Contains(error));
	}
}

Let’s break down the code above. Firstly we are injecting an instance of the ScenarioContext. Specflow handles this for you so nothing more to do other than add it to the constructor. The context is then used to store the json body being passed in. The ‘When’ method is where we get busy. First we extract the previously stored request body and set up an HttpRequestMessage with a POST method and point it to a relative url with the given operation (which we expect to be Client for this scenario). We then create an HttpClient with which we send the request before disseminating the response into its various parts and storing back into the scenario context.

The final two methods are our asserts – firstly on the response status code and reason and finally on ensuring we get back the expected error within the response body.

Before I move on with test execution, I need to bring up the issue here of using the relative url. The test is not run in the context of a web api app so there is nothing there to call and I would expect to receive a 404 NotFound response. I could use an absolute url but then I would need to ensure it worked in all environments and also that it was up and running. Not ideal for automated pipelines.

WebApplicationFactory

WebApplicationFactory is a construct from Microsoft that creates a TestServer that can be used to fire api calls to with no dependency on the browser or spinning up your app. Yaaaay! And it’s not difficult to setup. Add the Microsoft.AspNetCore.Mvc.Testing nuget package to your test project and then simply inject a typed instance of it and call its Create method to generate your HttpClient without having to worry about absolute urls. The type you typically use is the Startup class from your api:

[Binding]
public class ClientPostSteps
{
	private readonly ScenarioContext context;
	private readonly WebApplicationFactory<Startup> webApplicationFactory;

	public ClientPostSteps(
		ScenarioContext context,
		WebApplicationFactory<Startup> webApplicationFactory)
	{
		this.context = context;
		this.webApplicationFactory = webApplicationFactory;
	}

	[Given(@"I have the following request body:")]
	public void GivenIHaveTheFollowingRequestBody(string json)
	{
		// add the request into the scenario context for later use
		context.Set(json, "Request");
	}
	
	[When(@"I post this request to the ""(.*)"" operation")]
	public async Task WhenIPostThisRequestToTheOperation(string operation)
	{
		// retrieve request
		var requestBody = context.Get<string>("Request");

		// set up Http Request Message
		var request = new HttpRequestMessage(HttpMethod.Post, $"/{operation}")
		{
			Content = new StringContent(requestBody)
			{
				Headers = 
				{ 
					ContentType = new MediaTypeHeaderValue("application/json")
				}
			}
		};

		// create an http client
		var client = webApplicationFactory.CreateClient();

		// let's post
		var response = await client.SendAsync(request).ConfigureAwait(false);

		try
		{
			context.Set(response.StatusCode, "ResponseStatusCode");
			context.Set(response.ReasonPhrase, "ResponseReasonPhrase");
			var responseBody = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
			context.Set(responseBody, "ResponseBody");
		}
		finally
		{
			// move along, move along
		}
	}
	
	[Then(@"the result should be a (.*) \(""(.*)""\) response")]
	public void ThenTheResultShouldBeAResponse(int statusCode, string reasonPhrase)
	{
		Assert.Equal(statusCode, (int)context.Get<HttpStatusCode>("ResponseStatusCode"));
		Assert.Equal(reasonPhrase, context.Get<string>("ResponseReasonPhrase"));
	}
	
	[Then(@"the response body description should include the following text: ""(.*)""")]
	public void ThenTheResponseBodyDescriptionShouldIncludeTheFollowingText(string error)
	{
		Assert.True(context.Get<string>("ResponseBody").Contains(error));
	}
}

This then allows you to spin up your test on your dev machine, within a CI/CD pipeline, wherever you have dotnet core installed and run it. You can kick it off as you do any other unit test.

One of the big benefits of this is, having set up your steps to use parameters where it makes sense, you can now add in new scenarios without having to change the code in your steps code file:

These feature files, coupled with the WebApplicationFactory not only allow for automated tests to be run in your devops pipeline with no test harness required; they can also relate directly back to business requirements in a way that can be understood and validated by Product Owners or Business Analysts. That can only be a good thing! 🙂