# Better Protobuf / gRPC Support for Python
![](https://github.com/danielgtaylor/python-asertoproto/workflows/CI/badge.svg)
> :octocat: If you're reading this on github, please be aware that it might mention unreleased features! See the latest released README on [pypi](https://pypi.org/project/asertoproto/).
This project aims to provide an improved experience when using Protobuf / gRPC in a modern Python environment by making use of modern language features and generating readable, understandable, idiomatic Python code. It will not support legacy features or environments (e.g. Protobuf 2). The following are supported:
- Protobuf 3 & gRPC code generation
- Both binary & JSON serialization is built-in
- Python 3.6+ making use of:
- Enums
- Dataclasses
- `async`/`await`
- Timezone-aware `datetime` and `timedelta` objects
- Relative imports
- Mypy type checking
This project is heavily inspired by, and borrows functionality from:
- https://github.com/protocolbuffers/protobuf/tree/master/python
- https://github.com/eigenein/protobuf/
- https://github.com/vmagamedov/grpclib
## Motivation
This project exists because I am unhappy with the state of the official Google protoc plugin for Python.
- No `async` support (requires additional `grpclib` plugin)
- No typing support or code completion/intelligence (requires additional `mypy` plugin)
- No `__init__.py` module files get generated
- Output is not importable
- Import paths break in Python 3 unless you mess with `sys.path`
- Bugs when names clash (e.g. `codecs` package)
- Generated code is not idiomatic
- Completely unreadable runtime code-generation
- Much code looks like C++ or Java ported 1:1 to Python
- Capitalized function names like `HasField()` and `SerializeToString()`
- Uses `SerializeToString()` rather than the built-in `__bytes__()`
- Special wrapped types don't use Python's `None`
- Timestamp/duration types don't use Python's built-in `datetime` module
This project is a reimplementation from the ground up focused on idiomatic modern Python to help fix some of the above. While it may not be a 1:1 drop-in replacement due to changed method names and call patterns, the wire format is identical.
## Installation
First, install the package. Note that the `[compiler]` feature flag tells it to install extra dependencies only needed by the `protoc` plugin:
```sh
# Install both the library and compiler
pip install "asertoproto[compiler]"
# Install just the library (to use the generated code output)
pip install asertoproto
```
*Betterproto* is under active development. To install the latest beta version, use `pip install --pre asertoproto`.
## Getting Started
### Compiling proto files
Now, given you installed the compiler and have a proto file, e.g `example.proto`:
```protobuf
syntax = "proto3";
package hello;
// Greeting represents a message you can tell a user.
message Greeting {
string message = 1;
}
```
You can run the following to invoke protoc directly:
```sh
mkdir lib
protoc -I . --python_asertoproto_out=lib example.proto
```
or run the following to invoke protoc via grpcio-tools:
```sh
pip install grpcio-tools
python -m grpc_tools.protoc -I . --python_asertoproto_out=lib example.proto
```
This will generate `lib/hello/__init__.py` which looks like:
```python
# Generated by the protocol buffer compiler. DO NOT EDIT!
# sources: example.proto
# plugin: python-asertoproto
from dataclasses import dataclass
import asertoproto
@dataclass
class Greeting(asertoproto.Message):
"""Greeting represents a message you can tell a user."""
message: str = asertoproto.string_field(1)
```
Now you can use it!
```python
>>> from lib.hello import Greeting
>>> test = Greeting()
>>> test
Greeting(message='')
>>> test.message = "Hey!"
>>> test
Greeting(message="Hey!")
>>> serialized = bytes(test)
>>> serialized
b'\n\x04Hey!'
>>> another = Greeting().parse(serialized)
>>> another
Greeting(message="Hey!")
>>> another.to_dict()
{"message": "Hey!"}
>>> another.to_json(indent=2)
'{\n "message": "Hey!"\n}'
```
### Async gRPC Support
The generated Protobuf `Message` classes are compatible with [grpclib](https://github.com/vmagamedov/grpclib) so you are free to use it if you like. That said, this project also includes support for async gRPC stub generation with better static type checking and code completion support. It is enabled by default.
Given an example service definition:
```protobuf
syntax = "proto3";
package echo;
message EchoRequest {
string value = 1;
// Number of extra times to echo
uint32 extra_times = 2;
}
message EchoResponse {
repeated string values = 1;
}
message EchoStreamResponse {
string value = 1;
}
service Echo {
rpc Echo(EchoRequest) returns (EchoResponse);
rpc EchoStream(EchoRequest) returns (stream EchoStreamResponse);
}
```
Generate echo proto file:
```
python -m grpc_tools.protoc -I . --python_asertoproto_out=. echo.proto
```
A client can be implemented as follows:
```python
import asyncio
import echo
from grpclib.client import Channel
async def main():
channel = Channel(host="127.0.0.1", port=50051)
service = echo.EchoStub(channel)
response = await service.echo(value="hello", extra_times=1)
print(response)
async for response in service.echo_stream(value="hello", extra_times=1):
print(response)
# don't forget to close the channel when done!
channel.close()
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
```
which would output
```python
EchoResponse(values=['hello', 'hello'])
EchoStreamResponse(value='hello')
EchoStreamResponse(value='hello')
```
This project also produces server-facing stubs that can be used to implement a Python
gRPC server.
To use them, simply subclass the base class in the generated files and override the
service methods:
```python
import asyncio
from echo import EchoBase, EchoResponse, EchoStreamResponse
from grpclib.server import Server
from typing import AsyncIterator
class EchoService(EchoBase):
async def echo(self, value: str, extra_times: int) -> "EchoResponse":
return EchoResponse([value for _ in range(extra_times)])
async def echo_stream(self, value: str, extra_times: int) -> AsyncIterator["EchoStreamResponse"]:
for _ in range(extra_times):
yield EchoStreamResponse(value)
async def main():
server = Server([EchoService()])
await server.start("127.0.0.1", 50051)
await server.wait_closed()
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
```
### JSON
Both serializing and parsing are supported to/from JSON and Python dictionaries using the following methods:
- Dicts: `Message().to_dict()`, `Message().from_dict(...)`
- JSON: `Message().to_json()`, `Message().from_json(...)`
For compatibility the default is to convert field names to `camelCase`. You can control this behavior by passing a casing value, e.g:
```python
MyMessage().to_dict(casing=asertoproto.Casing.SNAKE)
```
### Determining if a message was sent
Sometimes it is useful to be able to determine whether a message has been sent on the wire. This is how the Google wrapper types work to let you know whether a value is unset, set as the default (zero value), or set as something else, for example.
Use `asertoproto.serialized_on_wire(message)` to determine if it was sent. This is a little bit different from the official Google generated Python code, and it lives outside the generated `Message` class to prevent name clashes. Note that it **only** supports Proto 3 and thus can **only** be used to check if `Message` fields are set. You cannot check if a scalar was sent on the wire.
```py
# Old way (official Google Protobuf package)
>>> mymessage.HasField('myfield')
# New way (this project)
>>> asertoproto.serialized_on_wire(mymessage.myfield)
```
### One-of Support
Protobuf supports grouping fields in a `oneof` clause. Only o