WireMock.Net : gRPC / ProtoBuf Support

⏩Intro

As author from WireMock.Net, I started this blog series where I’ll explain new features or advanced functionality and usage scenarios for WireMock.Net (a flexible product for stubbing and mocking web HTTP responses using advanced request matching and response templating).

My previous blogs are :

In this blog, I will zoom into the new gRPC and ProtoBuf support.

1️⃣ Content

This blog contains the several chapters which explain gRPC and why and how I did update WireMock.Net and I also walk you through the technical details.

In case you just want to know to use this new functionality, you can jump directly to 4️⃣ Using the new gRPC functionality.

2️⃣ gRPC

gRPC, or RPC (Remote Procedure Calls), is an open-source communication protocol developed by Google, designed to optimize communication between microservices. Its growing popularity can be attributed to several key features:

Efficiency: gRPC uses Protocol Buffers, a language-neutral, platform-neutral, extensible mechanism for serializing structured data, which is more efficient and faster than traditional text-based formats like JSON.
Language Agnosticity: gRPC supports a multitude of programming languages, allowing seamless integration across various parts of a system that might be written in different languages.
Bi-directional : gRPC supports bi-directional , providing a more dynamic communication model compared to traditional request/response architectures.
Deadline/Timeouts and Cancellation: It natively supports specifying deadlines and handling request cancellations, which is crucial for building responsive systems.
Strongly-typed Contracts: gRPC enforces contracts between client and server via Protocol Buffers, reducing the likelihood of communication issues caused by data format errors.

The integration of gRPC in WireMock.Net, is very useful for several reasons:

Testing Microservices: With the rise of microservices architecture, having a tool that supports gRPC allows developers to efficiently test services which communicate over gRPC.
Consistency in Testing Approaches: Teams can use a consistent approach for testing both HTTP (REST, GraphQL, …) and gRPC services, streamlining the testing process by providing the same usage which will decrease the learning curve for new developers.
Improved Quality Assurance: The ability to mock gRPC services allows for more comprehensive and reliable testing, leading to higher quality software.

In conclusion, the support for gRPC in WireMock.Net is a valuable addition, especially now where microservices and efficient communication protocols like gRPC are becoming more important. It not only simplifies the testing process for gRPC services but also ensures a more robust and consistent approach to testing across different types of services (like REST or GraphQL).

See also this blog (in Dutch) on using gRPC in .NET.

3️⃣ gRPC Support

In previous versions (< 1.5.47), only HTTP/1.1 request and response messages were implemented in WireMock.Net, so gRPC functionality using the HTTP/2 protocol was not yet supported.
Note that there is a side project GRPC-Mock-Server which enables gRPC support, however using this approach requires a separate product which makes it more difficult to integrate and use this in your existing integration tests using WireMock.Net.

This is the reason I investigated the possibility to extend WireMock.Net to add native support for gRPC. This would require doing the following tasks:

Get generic understanding on ProtoBuf (Protocol Buffers).
Be able to parse gRPC ProtoBuf messages and build a matcher which can easily match a ProtoBuf request-message using the JSON syntax.
Be able to generate a gRPC ProtoBuf response message using the JSON syntax. In addition, response templating using Handlebars should also work for gRPC ProtoBuf response messages.
HTTP/2 support must be added. This includes support for “Trailing Headers”.
Last, but not least: the fluent-interface builder for this new functionality must be easy to understand and to use.

3.1 Generic understanding on ProtoBuf

Protocol Buffers are language-neutral, platform-neutral extensible mechanisms for serializing structured data. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

The Proto definition describes how the data should be structured.
An example for a Person could be like:

message person {
  string name = 1;
  int32 id = 2;
  string email = 3;
}

Here’s a breakdown of what each part means:

message person { ... } This defines a new message type called person. In ProtoBuf, a message is a structure that contains a set of typed fields. Messages are similar to classes in OOP languages .
string name = 1; This line declares a field called name of type string. The = 1 at the end defines the field’s tag number, which is used to identify this field in the binary representation.
int32 id = 2; Similar to the name field, this declares an id field of type int32 (a 32-bit integer).
string email = 3; This line declares another field called email, also of type string. Its tag number is 3, indicating its unique position within the person message.

Protocol Buffers encodes the data to a wire format which defines the details of how your message is sent on the “wire“.
Note that the ProtoBuf Definition is required in order to encode / decode the wire format because the field-names are not stored, only the position and the value are stored.

The Person message with the following data:

{
  "name": "stef heyenrath",
  "id": 42,
  "email": "[email protected]"
}

Is encoded to this C# byte stream:

var wireformat = new byte[]
{
    0x0A, 0x0E, 0x73, 0x74, 0x65, 0x66, 0x20, 0x68,
    0x65, 0x79, 0x65, 0x6E, 0x72, 0x61, 0x74, 0x68,
    0x10, 0x2A, 0x1A, 0x0E, 0x69, 0x6E, 0x66, 0x6F,
    0x40, 0x6D, 0x73, 0x74, 0x61, 0x63, 0x6B, 0x2E,
    0x6E, 0x6C
};

A protocol buffer message is a series of key-value pairs. The binary version of a message just uses the field’s number as the key. The name and declared type for each field can only be determined on the decoding end by referencing the message type’s definition (i.e. the Proto Definition file).

When a message is encoded, each key-value pair is turned into a record consisting of the:

field number; the 1-based index of the field
a wire type; tells the parser how big the payload after it is
a payload; the content (string, int32, …)

For an in-depth article on how the wire format is encoded, see this web-page.

3.2 Parsing gRPC binary encoded ProtoBuf message

When a gRPC client calls WireMock.Net, the ProtoBuf message is just send as bytes. The ProtoDefinition is required to decode this.
For this process, I did analyze different existing NuGet packages which could do this job. I ended up using protobuf-net which is a contract based serializer for .NET that can (de)serialize the data encoded in ProtoBuf format based on a Proto Definition (.proto).

However, this protobuf-net library can only parse the Proto Definition and generate C# classes for the request and response messages.
Therefore I created a new solution which does the following steps:

Use protobuf-net to generate C# code (classes) using the Proto Definition.
For the person example above, the following C# class is generated by protobuf-net:
Then compile this C# code into a C# Type using Roslyn (e.g. CSharpCompilation).
This type is loaded into the current Assembly.
The ProtoBuf bytes, together with the Message-Type (in this case it’s “Person”) are used to create an instance of that generated Person object using protobuf-net.
At the end, this Person instance is converted to JSON using NewtonSoft JSON.
With this last step, it’s possible to use the JSON specific matchers from WireMock.Net to do complex request message matching for ProtoBuf messages.

See flow diagram below for better overview:

3.3 Generate a gRPC ProtoBuf message using JSON syntax

A gRPC client calling WireMock.Net expects a response message which is encoded as ProtoBuf bytes. To make this possible, the reverse from the step 3.2 should be implemented following these steps:

Use protobuf-net to generate C# code (classes) using the Proto Definition and compile this C# code into a type using Roslyn. Then load that type into the current Assembly.
Convert the (anonymous) PersonResponse instance to a JSON string using NewtonSoft JSON. Optionally, it should be possible to use the Handlebars.NET transformer.
Deserialize this JSON string using the correct PersonResponse type to an PersonResponse instance.
Use protobuf-net to serialize the PersonResponse (using the Message-Type) to generate a ProtoBuf byte array
WireMock.Net returns that ProtoBuf byte array to the calling gRPC client

See flow diagram below for more detail:

I’ve integrated all the above functionality in a project named ProtoBufJsonConverter.

And an online tool for encoding and decoding ProtoBuf messages to JSON is: WebAssembly Blazor App : ProtoBuf encoder / decoder.

3.4 HTTP/2 and support for Trailing Headers

Trailing headers in gRPC and HTTP/2 play a crucial role in optimizing communication and data transmission. They are used for several important reasons:

Efficient Transmission of Metadata: Trailing headers are used to send metadata that is only known at the end of a message stream. In the context of HTTP/2 and gRPC, this is particularly useful for status codes and status messages, which are determined after the full request or response body has been processed.
Support for : gRPC, built on top of HTTP/2, heavily relies on capabilities. Trailing headers are essential in scenarios where the size or the complete set of data is not known upfront. They allow the transmission of additional information or summaries upon the completion of the data stream. Note that is not yet supported by WireMock.Net.
Error Handling and Status Codes: In gRPC, trailing headers are often used to send status codes and error messages related to the request. This is crucial for client-side applications to understand the result of the request, especially in cases of failure or partial success.
Data Integrity and Confirmation: Trailing headers can be used to send checksums or hashes to confirm the integrity of the transmitted data. This ensures that the data received is complete and unaltered, which is important for reliability and security.
Compliance with HTTP/2 Protocol: HTTP/2 introduced the concept of trailing headers as a part of its specification. gRPC, being built on top of HTTP/2, inherits and utilizes this feature for efficient communication and adherence to the underlying protocol standards.

Both HTTP/2 and Trailing headers are supported on .NET Core 3.0 & .NET Standard 2.0 and later versions and also on .NET Framework 4.6.2 and later, so this new functionality will only work if the UnitTest or StandAlone application is using compatible .NET frameworks.
Note that the Docker and TestContainer versions will be fine because these target .NET 6.

HTTP/2

Listening to HTTP/2 requests in a .NET Core application involves setting up a web host with HTTP/2 support. This is straightforward using the the built-in Kestrel web server in ASP.NET Core, which supports HTTP/2.
The following code example shows how to enable HTTP/2:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup();
            webBuilder.ConfigureKestrel(options =>
            {
                // Set up HTTP/2
                options.ListenAnyIP(5001, listenOptions =>
                {
                    listenOptions.Protocols = HttpProtocols.Http2;
                });
            });
        });

For easy use in WireMock.Net, I’ve added the possibility to use a specific URL prefix, like:

grpc:// (normal connection)
grpcs:// (for secure connection)

Example:

var urlGrpc = "grpc://localhost:9093/";
var urlGrpcSecure = "grpcs://localhost:9094/";

var server = WireMockServer.Start(new WireMockServerSettings
{
    Urls = new[] { urlGrpc, urlGrpcSecure },

    // Other options ...
});

Trailing Headers

Sending Trailing headers is straightforward. Use the following code in an API controller:

[ApiController]
[Route("[controller]")]
public class SampleController : ControllerBase
{
    [HttpGet]
    public IActionResult Get()
    {
        if (Response.SupportsTrailers())
        {
            // Append a trailing header
            Response.AppendTrailer("Custom-Trailer-Header", "HeaderValue");
        }

        return Ok();
    }
}

Some examples of common gRPC trailing headers and their meanings:

grpc-status: This is perhaps the most important trailing header in gRPC. It communicates the status of the gRPC call. It’s a integer value representing various statuses like OK, Internal Error, Unimplemented, etc. For example, a grpc-status of 0 indicates a successful call. See this web-page for a list from all status codes.
grpc-message: Accompanies the grpc-status header. It provides a human-readable description or additional information about the status of the gRPC call, especially in cases of errors.
For instance, if grpc-status is non-zero, grpc-message might contain a description of the error.

4️⃣ Using the new gRPC functionality

This chapter explains how to use this new functionality in your WireMock.Net enabled unit-tests:

4.1 Define a Proto Definition file (greet.proto)

syntax = "proto3";
 
package greet;
 
// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply);
}
 
// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}
 
// The response message containing the greetings
message HelloReply {
  string message = 1;
}

4.2 Start the WireMock.Net server and define the mapping for the Grpc call:

// Read the 'greet.proto' ProtoDefinition file as text and store it in a variable
var protoDefinitionText = File.ReadAllText(@"c:\grpc\greet.proto");

// Define an unique identifier for that ProtoDefinition to make it possible to refer
// to that ProtoDefinition in the different mappings
var protoDefinitionId = "GrpcGreet";

// Start the WireMockServer and enable HTTP/2 support
var server = WireMockServer.Start(useHttp2: true);

server
  // Now call the new AddProtoDefinition method to register this identifier
  // and ProtoDefinition in WireMock.Net
  .AddProtoDefinition(protoDefinitionId, protoDefinitionText)

  // Define the Request matching logic which means in this case:
  // - Match on HTTP POST
  // - Match when the client calls the SayHello method on the Greeter-service
  // - Use a JsonMatcher so that this request is only mapped when the name
  //   equals "stef".
  .Given(Request.Create()
    .UsingPost()
    .WithPath("/grpc/greet.Greeter/SayHello")
    .WitdyAsProtoBuf("greet.HelloRequest", new JsonMatcher(new { name = "stef" }))
  )

  // Define that this mapping should use the specified protoDefinitionId for both 
  // the Request and the Response
  .WithProtoDefinition(protoDefinitionId)

  // Build a response which will:
  // - Return the correct Content-Type header and Grpc Trailing header
  // - Define the response as an anonymous object and use the Handlebars 
  //   Transformer to return a personalized message
  // - Return a ProtoBuf byte-array response using the HelloReply method
  .RespondWith(Response.Create()
    .WithHeader("Content-Type", "application/grpc")
    .WithTrailingHeader("grpc-status", "0")
    .WitdyAsProtoBuf("greet.HelloReply",
    new
    {
      message = "hello {{request.BodyAsJson.name}} {{request.method}}"
    })
    .WithTransformer()
  );

4.3 Testing the endpoint in PostMan

When all is configured correctly and WireMock server is running, you can use the new Grpc functionality in PostMan to create and execute a Grpc ProtoBuf request.

The url
The gRPC Service (Greeter) and the Method (SayHello)
The request message (as json)
The response message (as json)
The Trailing Header (grpc-status = 0 = OK)

4.4 Unit Testing

Add the proto file and the required NuGet packages to the unit-test csproj file:

<ItemGroup>
  <PackageReference Include="Google.Protobuf" Version="3.25.1"/>
  <PackageReference Include="Grpc.Net.Client" Version="2.59.0"/>
  <PackageReference Include="Grpc.Tools" Version="2.60.0">
    <PrivateAssets>all</PrivateAssets>
    <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
  </PackageReference>
  <Protobuf Include="Grpc\greet.proto" GrpcServices="Client"/>
</ItemGroup>

A complete unit test which uses a gRPC client (generated by Grpc-Tools) to call a http2 endpoint on WireMock.Net looks like this:

// Arrange
var id = $"test-{Guid.NewGuid()}";

using var server = WireMockServer.Start(useHttp2: true);

server
  .AddProtoDefinition(id, ProtoDefinition)
  .Given(Request.Create()
    .UsingPost()
    .WithHttpVersion("2")
    .WithPath("/greet.Greeter/SayHello")
    .WitdyAsProtoBuf("greet.HelloRequest", new JsonMatcher(new { name = "stef" }))
  )
  .WithProtoDefinition(id)
  .RespondWith(Response.Create()
    .WithHeader("Content-Type", "application/grpc")
    .WithTrailingHeader("grpc-status", "0")
    .WitdyAsProtoBuf("greet.HelloReply",
      new
      {
        message = "hi {{request.BodyAsJson.name}}"
      }
    )
    .WithTransformer()
  );

// Act
var channel = GrpcChannel.ForAddress(server.Url!);

var client = new Greeter.GreeterClient(channel);

var reply = await client.SayHelloAsync(new HelloRequest { Name = "stef" });

// Assert
reply.Message.Should().Be("hi stef");

server.Stop();

5️⃣ Conclusion

New functionality

This new functionality extends WireMock.Net even more and makes it very easy to write unit-integrations-tests for your gRPC enabled client application for mocking gRPC requests and responses.

Future

At this moment, only unary request-response gRPC messages are supported, support like Client Side , Service Side or Bidirectional is not yet supported. Maybe this will be added later.

❔ Feedback or Questions

Questions and issues can also be added on my -project.
Do you have any feedback or questions regarding this or one of my other blogs? Feel free to contact me!

📝Notes

Some content in this blog is created with the help of an AI. I did review and revise the content were needed.

📚 Additional resources

Written by: Stef Heyenrath

Stef started writing software for the Microsoft .NET framework in 2007. Over the years, he has developed into a Microsoft specialist with experience in: backend technologies such as .NET, NETStandard, ASP.NET, Ethereum, Azure, and other cloud providers. In addition he worked with several frontend technologies such as Blazor, React, Angular, Vue.js.

He is the author from WireMock.Net.

Mission: Writing quality and structured software with passion in a scrum team for technically challenging projects.

Want to know more about our experts? Contact us!

Neem contact op Stef op LinkedIn

Cookie	Duur	Beschrijving
cookielawinfo-checkbox-advertisement	1 year	Set by the GDPR Cookie Consent plugin, this cookie records the user consent for the cookies in the "Advertisement" category.
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
CookieLawInfoConsent	1 year	CookieYes sets this cookie to record the default button state of the corresponding category and the status of CCPA. It works only in coordination with the primary cookie.
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Cookie	Duur	Beschrijving
yt-player-bandwidth	never	The yt-player-bandwidth cookie is used to store the user's video player preferences and settings, particularly related to bandwidth and quality on YouTube.
yt-player-headers-readable	never	The yt-player-headers-readable cookie is used by YouTube to store user preferences related to video playback and interface, enhancing the user's viewing experience.
yt-remote-cast-available	session	The yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.
yt-remote-cast-installed	session	The yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.
yt-remote-connected-devices	never	YouTube sets this cookie to store the user's video preferences using embedded YouTube videos.
yt-remote-device-id	never	YouTube sets this cookie to store the user's video preferences using embedded YouTube videos.
yt-remote-fast-check-period	session	The yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.
yt-remote-session-app	session	The yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.
yt-remote-session-name	session	The yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.
ytidb::LAST_RESULT_ENTRY_KEY	never	The cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.

Cookie	Duur	Beschrijving
IDE	1 year 24 days	Google DoubleClick IDE cookies store information about how the user uses the website to present them with relevant ads according to the user profile.
PREF	8 months	PREF cookie is set by Youtube to store user preferences like language, format of search results and other customizations for YouTube Videos embedded in different sites.
test_cookie	15 minutes	doubleclick.net sets this cookie to determine if the user's browser supports cookies.
VISITOR_INFO1_LIVE	6 months	YouTube sets this cookie to measure bandwidth, determining whether the user gets the new or old player interface.
VISITOR_PRIVACY_METADATA	1 year 1 month	YouTube sets this cookie to store the user's cookie consent state for the current domain.
YSC	session	Youtube sets this cookie to track the views of embedded videos on Youtube pages.
yt.innertube::nextId	never	YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.
yt.innertube::requests	never	YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.

Blog

WireMock.Net : gRPC / ProtoBuf Support

⏩Intro

1️⃣ Content

2️⃣ gRPC

3️⃣ gRPC Support

3.1 Generic understanding on ProtoBuf

3.2 Parsing gRPC binary encoded ProtoBuf message

3.3 Generate a gRPC ProtoBuf message using JSON syntax

3.4 HTTP/2 and support for Trailing Headers

HTTP/2

Trailing Headers

4️⃣ Using the new gRPC functionality

4.1 Define a Proto Definition file (greet.proto)

4.2 Start the WireMock.Net server and define the mapping for the Grpc call:

4.3 Testing the endpoint in PostMan

4.4 Unit Testing

5️⃣ Conclusion

New functionality

Future

❔ Feedback or Questions

📝Notes

📚 Additional resources

Stef Heyenrath

Written by: Stef Heyenrath

Over mstack

Links

Hoofdkantoor

Blog

WireMock.Net : gRPC / ProtoBuf Support

⏩Intro

1️⃣ Content

2️⃣ gRPC

3️⃣ gRPC Support

3.1 Generic understanding on ProtoBuf

3.2 Parsing gRPC binary encoded ProtoBuf message

3.3 Generate a gRPC ProtoBuf message using JSON syntax

3.4 HTTP/2 and support for Trailing Headers

HTTP/2

Trailing Headers

4️⃣ Using the new gRPC functionality

4.1 Define a Proto Definition file (greet.proto)

4.2 Start the WireMock.Net server and define the mapping for the Grpc call:

4.3 Testing the endpoint in PostMan

4.4 Unit Testing

5️⃣ Conclusion

New functionality

Future

❔ Feedback or Questions

📝Notes

📚 Additional resources

Stef Heyenrath

Written by: Stef Heyenrath

Cookies