lyqtRpc

中文

A lightweight C++20 RPC framework built on muduo + Protobuf. Dual TCP / shared-memory zero-copy transport, etcd registry, circuit breaker, token bucket, distributed tracing, Prometheus metrics, and an HTTP-to-RPC API gateway — all in one repo.

Author: lczllx · Language: C++20 · Network: muduo · Transport: TCP & SHM zero-copy · Serialization: Protobuf, JSON, FlatBuffers · Build: CMake

Highlights

• Dual transport: TCP (LV variable-length framing) and SHM zero-copy (per-client ring buffer + Protobuf SerializeToArray direct write)
• Service governance: etcd registry, heartbeat keep-alive, CAS leader election, load reporting, three-state circuit breaker, token bucket rate limiting
• Call models: synchronous, std::future async, and callback-based
• Pluggable serialization: ISerializer abstraction, default Protobuf, JSON for debugging, FlatBuffers for zero-copy reads
• Distributed tracing: trace_id + span_id end-to-end propagation
• Multi-client concurrency: SHM path with muduo EventLoopThread worker pool, per-client independent ring buffers
• Observability: built-in Prometheus /metrics endpoint (text format 0.0.4), covering request count / latency histogram / concurrency / error count / connection count / circuit breaker state / token bucket / process-level metrics — mirrors brpc /vars
• API Gateway: HTTP→RPC gateway on the same muduo foundation, with route matching, rate limiting, circuit breaker, Prometheus metrics, and distributed tracing — deployed as a separate process, zero changes to the RPC framework

Performance: lyqtRpc vs brpc

Test environment: 4C8G cloud VM, Ubuntu 22.04, g++ 11.4.0, all Protobuf, echo payload. brpc 1.17.0.

Method: the brpc column takes the median of its per-second samples (-connection_type=single, pooled for the 4-thread row); that output carries only mean latency, so the brpc P99 cells are left blank. The lyqtRpc column is a 30-second steady-state run (TCP: the steady section of example/benchmark/run_benchmark.sh; SHM: example/shm/run_shm_benchmark.sh proto; override the duration with STEADY_SECS).

Single-thread latency & throughput

Payload	brpc TCP	lyqtRpc TCP Proto	lyqtRpc SHM Proto ZC
16B QPS	14,038	10,094	28,561
16B mean	69μs	98μs	—
16B P99	—	146μs	36μs
64KB QPS	4,241	2,209	12,888
64KB mean	227μs	444μs	—
64KB P99	—	707μs	120μs

4-thread concurrency

Metric	brpc TCP	lyqtRpc TCP Proto	lyqtRpc SHM Proto ZC
QPS	47,075	33,379	127,197
Mean	82μs	119μs	—
P99	—	217μs	52μs

SHM throughput is 2.0× brpc's TCP path at 16B single-thread, 3.0× at 64KB, and 2.7× at 4 threads. The TCP path reaches ≈72% of brpc at 16B single-thread and ≈52% at 64KB, stemming from bthread coroutines, IOBuf zero-copy chains, and baidu_std multiplexing in brpc.

Quick Start

Requirements: g++ 11 or newer (C++20), CMake ≥ 3.16.

git clone https://github.com/lczllx/lyqtRpc.git
cd lyqtRpc
git submodule update --init --recursive
bash autobuild/quick_build.sh

Docker

bash autobuild/docker.sh doctor
bash autobuild/docker.sh setup

Examples

# TCP RPC (requires etcd)
docker compose up -d etcd
bash demosh/demo.sh etcd

# SHM Proto zero-copy (no extra dependencies)
cd rpc/build
./bin/shm_proto_server &
./bin/shm_proto_client

# Full-path benchmark
cd example/shm && bash run_shm_benchmark.sh all

Prometheus metrics

cd rpc/build
./bin/benchmark_server 8889 0 8080 0 &
./bin/benchmark_client single echo 2000 1 1
curl localhost:9090/metrics

Add the address to prometheus.yml scrape_configs. Percentiles are computed query-side: histogram_quantile(0.99, rate(rpc_request_duration_us_bucket[1m])).

API Gateway

cd rpc/build
./bin/benchmark_server 8889 &      # 1. start RPC backend
./bin/gateway_server &             # 2. start gateway (HTTP :8080, metrics :9091)

curl -d '{"data":"hello"}' localhost:8080/api/echo   # → RPC echo backend
curl localhost:8080/api/health                        # → health check
curl localhost:8080/diagnose                          # → rate limiter + breaker status
curl localhost:9091/metrics | grep gateway            # → gateway metrics

Architecture

Provider ──REGISTER/HEARTBEAT──> Registry(etcd) <──DISCOVER── Consumer
   │                                                          │
   └── RPC call ────────────────────────────────────────────>─┘

• LV framing: | 4B total_len | 4B msg_type | 4B id_len | id | body |
• SHM zero-copy: SerializeToArray → ring buffer → eventfd → ParseFromArray, bypassing TCP
• Registry backend: LCZ_ETCD env var switches between Memory / Etcd; defaults to in-memory
• Circuit breaker: three-state (CLOSED→OPEN→HALF_OPEN), method×host granularity, memory/etcd persistence
• Rate limiter: TokenBucket + BACKOFF automatic backoff and retry
• Thread pool: muduo EventLoopThread, per-client ring buffer, lock-free SPSC

See docs/en/architecture-en.md for detailed flowcharts.

Directory

lyqtRpc/
├── rpc/
│   ├── src/
│   │   ├── client/           # RpcClient, ClientDiscover, CircuitBreaker, ShmClient
│   │   ├── server/           # RpcServer, Registry, LeaderElection, ShmServer
│   │   └── general/          # ShmChannel, LVProtocol, MessageFactory, Serializer, Logger
│   ├── tests/                # 189 GTest cases
│   ├── example/              # Examples + benchmarks
│   ├── proto/                # Protobuf definitions
│   └── muduo/                # Git submodule
├── gateway/
│   ├── src/                  # HttpServer, HttpRouter, GatewayHandler, DiagnoseHandler
│   └── example/              # gateway_server entry point
├── autobuild/                # Build + Docker scripts
├── demosh/                   # Demo scripts
├── docs/                     # Design docs (cn + en)
├── grafana/dashboards/       # Grafana dashboard JSON
├── Dockerfile
└── docker-compose.yml

Build

Two ways to build — system deps (fast) or vcpkg (zero-setup, cross-platform).

1. System deps (apt) — recommended

Dependencies (one apt install):

sudo apt-get install -y build-essential cmake g++ make \
  libboost-dev libjsoncpp-dev libcurl4-openssl-dev \
  protobuf-compiler libprotobuf-dev \
  flatbuffers-compiler libflatbuffers-dev    # optional, for SHM FlatBuf path

Compile (CMake presets):

cd lyqtRpc/rpc
cmake --preset system -B build -DLCZ_RPC_BUILD_EXAMPLES=ON
cmake --build build -j$(nproc)

Unit tests: add -DLCZ_RPC_BUILD_TESTS=ON (needs libgtest-dev), then run ./build/bin/lcz_rpc_unit_tests.

2. vcpkg (zero apt deps, cross-platform)

cd lyqtRpc/rpc
cmake --preset vcpkg -B build-vcpkg -DLCZ_RPC_BUILD_EXAMPLES=ON
cmake --build build-vcpkg -j$(nproc)

First build downloads and compiles all deps (protobuf / curl / jsoncpp / flatbuffers / gtest / benchmark / fmt) from source — ~30min. Subsequent builds are cached.

Roadmap

• GitHub Actions CI (build + unit tests + SHM smoke test)
• Docker image + docker-compose
• etcd registry backend (Lease + LCZ_ETCD env-var switch)
• Three-state circuit breaker (method×host, configurable)
• Registry multi-instance HA (etcd lease + CAS election)
• Token bucket rate limiting + BACKOFF
• Distributed tracing (trace_id/span_id)
• Pluggable serializer (ISerializer + ProtobufSerializer)
• Prometheus /metrics (QPS / latency histogram / error codes / process metrics)

Known Limitations

• TCP path reuses long-lived connections per host (connection pool), but each host still has a single connection — send() serializes behind a lock when threads share a host; protocol multiplexing (stream_id) still pending
• etcd heartbeat re-registers on every keepalive failure (lease TTL too short), write amplification under load
• SHM payloads >64KB copy twice through the ring buffer; throughput worse than TCP zero-copy equivalents
• No auth / encryption; no streaming RPC; Topic has no persistence
• Unit tests cover core modules, circuit breaker, registry, load balancing, retry/backoff, topic, and the SHM adaptor; the full TCP network layer is still untested