Skip to content

Add socket reserve/use connection affinity + fd_version primitives#3348

Closed
rajvarun77 wants to merge 1 commit into
apache:masterfrom
rajvarun77:socket-bind-affinity
Closed

Add socket reserve/use connection affinity + fd_version primitives#3348
rajvarun77 wants to merge 1 commit into
apache:masterfrom
rajvarun77:socket-bind-affinity

Conversation

@rajvarun77

Copy link
Copy Markdown
Contributor

Generic Controller/Socket plumbing that lets a protocol pin a pooled/short connection across multiple RPCs (connection affinity). This is the shared substrate extracted from the MySQL client PR #3330 so it can be reviewed independently; #3330 is now stacked on top of this branch and contains only the MySQL-specific code.

Controller

  • BindSockAction {NONE,RESERVE,USE} encoded in two Controller::_flags bits (FLAGS_BIND_SOCK_RESERVE/USE) via set_bind_sock_action()/bind_sock_action() — no dedicated member added to the hot Controller.
  • _bind_sock holds a socket reserved by a previous RPC. EndRPC propagates the action to the current Call; Call::OnComplete reserves the socket (on success) instead of returning it to the pool / failing it; IssueRPC reuses the reserved socket when the action is USE.
  • _session_data: opaque per-RPC slot a protocol codec may use to carry typed state between serialize/pack/parse (not owned by Controller).

Socket

  • _fd_version (relaxed atomic, bumped on every ResetFileDescriptor) + fd_version() accessor — an ABA guard so an affinity holder can detect that a reserved fd was reconnected underneath it. Atomic because it is written on the reconnect path (no socket lock held) and read from other threads.
  • Socket::Write accepts an empty buffer when WriteOptions::auth_flags is set, for server-greets-first handshakes that send their first bytes from the connection-phase handler rather than from data.

Backup/retry fix

Also fixes an uninitialized read this machinery would otherwise introduce: the Call(Call*) copy constructor used for backup requests and retries did not initialize bind_sock_action, so OnComplete could read indeterminate bits and (when they matched RESERVE/USE) divert the backup call's socket away from the pool-return path, hanging the RPC (manifested as ChannelTest.backup_request timing out). Initialized to BIND_SOCK_NONE, matching Call::Reset(); a backup/retry never inherits affinity.

No protocol uses these primitives in this PR; the first consumer is MySQL (#3330).

🤖 Generated with Claude Code

Introduces generic Controller/Socket plumbing that lets a protocol pin a
pooled/short connection across multiple RPCs (connection affinity), needed
by request/response protocols with per-connection server state such as MySQL
interactive transactions and prepared statements. No protocol uses it yet;
this is the shared substrate for the MySQL client PR (apache#3330), split out for
independent review.

Controller:
- BindSockAction {NONE,RESERVE,USE} encoded in two Controller::_flags bits
  (FLAGS_BIND_SOCK_RESERVE/USE) via set_bind_sock_action()/bind_sock_action();
  no dedicated member on the hot Controller.
- _bind_sock holds a socket reserved by a previous RPC; EndRPC propagates the
  action to the current Call, and Call::OnComplete reserves the socket (on
  success) instead of returning it to the pool / failing it. IssueRPC reuses
  the reserved socket when the action is USE.
- _session_data: opaque per-RPC slot a protocol codec may use to carry typed
  state between serialize/pack/parse (not owned by Controller).

Socket:
- _fd_version (relaxed atomic, bumped on every ResetFileDescriptor) plus a
  fd_version() accessor, an ABA guard so an affinity holder can detect that a
  reserved fd was reconnected underneath it. Atomic because it is written on
  the reconnect path (no socket lock held) and read from other threads.
- Socket::Write accepts an empty buffer when WriteOptions::auth_flags is set,
  for server-greets-first handshakes that send their first bytes from the
  connection-phase handler rather than from `data`.

Also fixes an uninitialized read this machinery would otherwise introduce:
the Call(Call*) copy constructor used for backup requests and retries did not
initialize bind_sock_action, so OnComplete could read indeterminate bits and
(when they matched RESERVE/USE) divert the backup call's socket away from the
pool-return path, hanging the RPC. Initialize to BIND_SOCK_NONE, matching
Call::Reset(); a backup/retry never inherits affinity.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@rajvarun77

Copy link
Copy Markdown
Contributor Author

This is the shared-core substrate extracted from #3330 (MySQL client protocol), which is now stacked on top of this PR. Reviewing this first keeps the generic Controller/Socket change small and self-contained.

@rajvarun77

Copy link
Copy Markdown
Contributor Author

Reverting the split — folding the backup/retry copy-constructor fix back into #3330 directly; a separate core PR isn't warranted for a one-line initialization fix.

@rajvarun77 rajvarun77 closed this Jun 15, 2026
@rajvarun77 rajvarun77 deleted the socket-bind-affinity branch June 15, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant