Nimendra's Notes 🪴

gRPC - Protobuf

6 min read

distributedsystem/microservice/grpc

Interface Definition Language (IDL)
[!video]- Protocol Buffers Crash Course


Protocol Buffers Documentation
🌟 Practical Protobuf - From Basic to Best Practices
🌟 How Protobuf Works—The Art of Data Encoding

Protocol buffers allow you to serialize structured data to be transmitted over a wire.

Info

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.
Protocol Buffers Documentation

Syntax

syntax = "proto3"
 
message CreateOrderRequest {
	int64 user_id = 1;
	repeated Item Items = 2;
	float amount = 3;
}

Fields

  • singular —> message can have at most one of the fields; this is default behaviour.
  • repeated —> May contain multiple values, including zero.

Field Types

👉 Language Guide (proto 3) | Protocol Buffers Documentation

  • For strings, the default value is the empty string.
  • For bytes, the default value is empty bytes.
  • For bools, the default value is false.
  • For numeric types, the default value is zero.
  • For message fields, the field is not set. Its exact value is language-dependent. See the generated code guide for details.
  • For enums, the default value is the first defined enum value, which must be 0. See Enum Default Value.
  • The default value for repeated fields is empty (generally an empty list in the appropriate language).
  • The default value for map fields is empty (generally an empty map in the appropriate language).

Field Names

Field Names are required and it should be lowercase and if it contains two or more words it should bet separated underscore(user_address_city).

Field Numbers and Reservations

Protobuf messages are serialized into a binary format that makes them compact and efficient. Each field in a message has a unique identifier (field number) that helps in serialization and deserialization.
When a message is converted to binary, it does **not store field names**, only the **field numbers and values**.

Why Field Numbers Shouldn’t Change ?

Since field numbers are used for identification in binary messages, changing them can break compatibility.

  • Suppose you had customer_id = 3; in an older message definition.
  • You later change 3 to user_id with a different type.
  • If an old client sends a message with customer_id, but the server now expects user_id, it will misinterpret the data.

This is why changing or reusing field numbers should be avoided.

Safely Remove Fields using reserved

If you need to remove a field, you should reserve its number and name. This prevents future developers from accidentally reusing them.
reserved field names cannot be reused or reassigned.

/* initial Definition */
message CreateOrderRequest {
  int64 customer_id = 3;  // This field will be removed in version 2
  repeated Item items = 4;
  float amount = 5;
}
/* Updated Definition */
message CreateOrderRequest {
  reserved 3;              // Prevents future reuse of field number 3
  reserved "customer_id";  // Prevents reuse of field name "customer_id"
 
  int64 user_id = 6;  // Introduced a new field instead of customer_id
  repeated Item items = 4;
  float amount = 5;
}

Performance

Use numbers 1–15 to minimize message size.

  • Field numbers 1–15 → Encoded in 1 byte (fast & efficient).
  • Field numbers 16–2047 → Encoded in 2 bytes.
  • Higher numbers require more bytes, increasing message size.

Using oneof for Exclusive Fields

The oneof feature in Protocol Buffers allows defining multiple fields where only one can be set at a time. This ensures mutual exclusivity without requiring extra logic in the implementation.

Example: Enforcing Exclusive Payment Methods

message CreatePaymentRequest {
  oneof payment_method {
    CreditCard credit_card = 1;
    PromoCode promo_code = 2;
  }
}

In this example, a request can contain either credit_card or promo_code, but not both.

Removing Fields from oneof (Backward Incompatibility)

  • If promo_code is removed from oneof and the server is upgraded, any older clients that send promo_code will lose that data.
  • This is considered a backward-incompatible change.
  • Clients using the previous version will send promo_code, but the server will discard the unknown field.

Updated Message (v2 - Breaking Change)

message CreatePaymentRequest {
  oneof payment_method {
    CreditCard credit_card = 1;
  }
}

📌 Impact: If a v1 client sends promo_code, the v2 server will not recognize it, leading to data loss.

Adding Fields to oneof (Forward Incompatibility)

  • If a new field is added to oneof, older clients will be unaware of this field.
  • Clients using an older version may not handle new fields properly, leading to forward-incompatible issues.

Updated Message (v3 - Breaking Change)

message CreatePaymentRequest {
  oneof payment_method {
    CreditCard credit_card = 1;
    BankTransfer bank_transfer = 3;
  }
}

📌 Impact: Older clients do not recognize bank_transfer and may not process requests correctly.

Moving Fields In/Out of oneof (Data Loss Risk)

Moving a field into or out of oneof can cause data loss, as previous messages could have had both fields set.

Before Moving (Non-exclusive promo_code)

message CreatePaymentRequest {
  oneof payment_method {
    CreditCard credit_card = 1;
  }
  PromoCode promo_code = 2;
}

After Moving (Exclusive promo_code)

message CreatePaymentRequest {
  oneof payment_method {
    CreditCard credit_card = 1;
    PromoCode promo_code = 2;
  }
}

📌 Impact: If a v1 client sends both credit_card and promo_code, only one will be preserved in v2, causing unexpected data loss.

Best Practices for Compatibility

  1. Avoid removing fields from oneof – If removal is necessary, consider introducing a new version of the message.
  2. Avoid moving fields in or out of oneof – This can cause loss of previously valid fields.
  3. Use semantic versioning (https://semver.org/) – Breaking changes should be indicated with a major version update.
  4. Communicate changes to API consumers – Ensure clients are aware of updates to prevent compatibility issues.

Predefined Protobuf Types

It is a standard Protobuf wrapper library(google.protobuf.wrappers.proto) that provides wrapper messages for primitive types (e.g., int, string, float). These wrappers allow you to handle optional values without default values.

protobuf/src/google/protobuf/wrappers.proto at e8edc5d5e72fa091b0086b4a6d12af0bb66d664b · protocolbuffers/protobuf · GitHub

  • Normally, primitive fields like string or int cannot be null in Protobuf.
  • But with wrapper types like google.protobuf.StringValue, fields can be nullable.

Why Use google.protobuf.StringValue?

In your service definition:

rpc getOrder(google.protobuf.StringValue) returns (Order);
  • Instead of using string, you are using google.protobuf.StringValue.
  • This means the request parameter can be empty (null), which is not possible with a regular string.

Equivalent Message Definition Without Wrapper

If you did not use google.protobuf.StringValue, you’d need to define a custom message like this:

message GetOrderRequest {
    string id = 1;
}

And update the RPC method:

rpc getOrder(GetOrderRequest) returns (Order);

Backlinks

Graph View