Skip to content

Update trace queue wait by timeout#3026

Merged
Kbhat1 merged 5 commits intomainfrom
fix-tracer-semaphore
Mar 6, 2026
Merged

Update trace queue wait by timeout#3026
Kbhat1 merged 5 commits intomainfrom
fix-tracer-semaphore

Conversation

@Kbhat1
Copy link
Contributor

@Kbhat1 Kbhat1 commented Mar 5, 2026

Describe your changes and provide context

  • Make trace RPC timeout apply to semaphore wait as well as trace execution
  • Prevent queued trace requests from hanging indefinitely by making semaphore acquisition context-aware.
  • `Add regression tests covering timeout while waiting for a slot and proper semaphore cleanup.

Testing performed to validate your change

  • Verifying on node

Start the trace timeout before semaphore acquisition and make the semaphore wait context-aware so queued trace requests fail on timeout or cancellation instead of hanging indefinitely.

Made-with: Cursor
@Kbhat1 Kbhat1 requested a review from yzang2019 March 5, 2026 19:44
@github-actions
Copy link

github-actions bot commented Mar 5, 2026

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedMar 6, 2026, 10:27 PM

@codecov
Copy link

codecov bot commented Mar 5, 2026

Codecov Report

❌ Patch coverage is 76.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.41%. Comparing base (f5844b5) to head (54fc974).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
evmrpc/tracers.go 76.00% 5 Missing and 7 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3026      +/-   ##
==========================================
+ Coverage   58.30%   58.41%   +0.11%     
==========================================
  Files        2079     2110      +31     
  Lines      171705   174882    +3177     
==========================================
+ Hits       100105   102165    +2060     
- Misses      62665    63696    +1031     
- Partials     8935     9021      +86     
Flag Coverage Δ
sei-chain-pr 67.22% <76.00%> (?)
sei-db 70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
evmrpc/tracers.go 66.50% <76.00%> (-2.57%) ⬇️

... and 88 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Kbhat1 Kbhat1 requested a review from blindchaser March 5, 2026 23:00
return func() { <-api.traceCallSemaphore }
select {
case api.traceCallSemaphore <- struct{}{}:
return func() { <-api.traceCallSemaphore }, nil
Copy link
Collaborator

@masih masih Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check if the context is not cancelled again here before returning. It is possible that we end up here after context cancellation since there is no guaranteed ordering for select statements in go.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, added a second context check after acquiring the slot so canceled requests do not slip through that race

Copy link
Collaborator

@masih masih Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Kbhat1 That change works but I think a simple if statement to check of ctx.Err is nil would be better. something like this:

if ctx.Err() != nil {
  <-api.traceCallSemaphore
  return func() {}, ctx.Err()
}
return func() { <-api.traceCallSemaphore }, nil

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

Release a trace semaphore slot if context cancellation wins the race with acquisition so timed out requests do not briefly consume a trace worker.

Made-with: Cursor
Reject trace requests immediately when all trace slots are occupied while still releasing the slot if cancellation wins the acquisition race.

Made-with: Cursor
@Kbhat1 Kbhat1 merged commit b424748 into main Mar 6, 2026
40 checks passed
@Kbhat1 Kbhat1 deleted the fix-tracer-semaphore branch March 6, 2026 21:31
@philipsu522
Copy link
Contributor

/backport

github-actions bot pushed a commit that referenced this pull request Mar 6, 2026
## Describe your changes and provide context
- Make trace RPC timeout apply to semaphore wait as well as trace
execution
- Prevent queued trace requests from hanging indefinitely by making
semaphore acquisition context-aware.
- `Add regression tests covering timeout while waiting for a slot and
proper semaphore cleanup.

## Testing performed to validate your change
- Verifying on node

(cherry picked from commit b424748)
@seidroid
Copy link

seidroid bot commented Mar 6, 2026

Successfully created backport PR for release/v6.3:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants