Skip to content

🐛 Truncate PageView fields to their column lengths - #213

Merged
ker0x merged 1 commit into
mainfrom
fix/issue-000-page-view-column-overflow
Aug 12, 2026
Merged

🐛 Truncate PageView fields to their column lengths#213
ker0x merged 1 commit into
mainfrom
fix/issue-000-page-view-column-overflow

Conversation

@ker0x

@ker0x ker0x commented Aug 12, 2026

Copy link
Copy Markdown
Member

Problem

Production was logging:

Error thrown while handling message App\Analytics\Application\Message\ProcessTrafficMessage. Removing from transport after 3 retries. Error: SQLSTATE[22001]: String data, right truncated: 7 ERROR: value too long for type character varying(255)

ProcessTrafficMessageHandler builds a PageView straight from the message payload, and every string in that payload comes from the raw HTTP request via TrafficDetector::getResult() — path info, SERVER_NAME, and the User-Agent / Referer headers. None were length-checked, but all five columns are varchar(255) (method is varchar(10)).

A scanner hitting a long URL, or any client sending an oversized User-Agent or Referer, produced an insert PostgreSQL rejected. The message then burned its 3 retries and was dropped, so each occurrence silently lost a page view.

The same hole existed in ProcessDailyTrafficMessageHandler, which builds PageView from parsed log entries with no length guard either.

Fix

PageView now enforces its own column widths in the constructor, so both producers are covered by one change:

  • Column lengths became constants, referenced by the #[Orm\Column] attributes and the truncation — the mapping and the guard can't drift apart.
  • Each string is cut with mb_substr(..., 'UTF-8'). Code points are the unit PostgreSQL counts for character varying, so a 255-code-point value always fits. s()->slice() was deliberately avoided: UnicodeString slices on grapheme clusters, and an emoji-heavy user agent could still overflow the column after a 255-grapheme slice.
  • referer keeps its null.

Why truncate rather than widen the columns

Request URLs have no upper bound, and url carries a btree index (url_idx) — PostgreSQL caps index entries at ~2704 bytes, so TEXT would just trade this error for index row size exceeds maximum. Losing the tail of an outlier URL is the right trade against losing the page view.

No migration neededdoctrine:schema:update --dump-sql reports the schema in sync, confirming the constants resolve identically inside the attributes.

Tests

New tests/Unit/Analytics/Domain/Entity/PageViewTest.php covers pass-through of values that fit, truncation of all six string fields, code-point-vs-byte cutting, and the null referer.

  • Full suite: 118 tests, 440 assertions, green
  • PHPStan level 8: clean
  • Doctrine mapping: valid

Note

Messages that already failed were removed from the transport after exhausting their retries, so this prevents recurrence — it does not recover the page views already lost.

Page views were built straight from raw request data (path info,
SERVER_NAME, User-Agent and Referer headers) with no length guard, while
every one of those columns is varchar(255). An over-long URL or header
made the insert fail with SQLSTATE[22001], and ProcessTrafficMessage was
dropped after exhausting its retries, silently losing the page view.

Enforce the column widths in the entity constructor so both producers
(ProcessTrafficMessageHandler and ProcessDailyTrafficMessageHandler) are
covered, with the lengths held in constants shared by the Doctrine
mapping and the truncation so the two cannot drift apart.

Cut with mb_substr on code points, the unit PostgreSQL counts for
character varying: UnicodeString::slice() works on grapheme clusters and
could still overflow the column.

Truncating beats widening the columns here, as request URLs have no upper
bound and url carries a btree index, which PostgreSQL caps at ~2704
bytes. The schema is unchanged, so no migration is needed.
@ker0x
ker0x merged commit 58c5390 into main Aug 12, 2026
1 check passed
@ker0x
ker0x deleted the fix/issue-000-page-view-column-overflow branch August 12, 2026 06:51
ker0x added a commit that referenced this pull request Aug 14, 2026
…-column-overflow

🐛 Truncate PageView fields to their column lengths
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant