🐛 Truncate PageView fields to their column lengths - #213
Merged
Conversation
Page views were built straight from raw request data (path info, SERVER_NAME, User-Agent and Referer headers) with no length guard, while every one of those columns is varchar(255). An over-long URL or header made the insert fail with SQLSTATE[22001], and ProcessTrafficMessage was dropped after exhausting its retries, silently losing the page view. Enforce the column widths in the entity constructor so both producers (ProcessTrafficMessageHandler and ProcessDailyTrafficMessageHandler) are covered, with the lengths held in constants shared by the Doctrine mapping and the truncation so the two cannot drift apart. Cut with mb_substr on code points, the unit PostgreSQL counts for character varying: UnicodeString::slice() works on grapheme clusters and could still overflow the column. Truncating beats widening the columns here, as request URLs have no upper bound and url carries a btree index, which PostgreSQL caps at ~2704 bytes. The schema is unchanged, so no migration is needed.
ker0x
added a commit
that referenced
this pull request
Aug 14, 2026
…-column-overflow 🐛 Truncate PageView fields to their column lengths
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Production was logging:
ProcessTrafficMessageHandlerbuilds aPageViewstraight from the message payload, and every string in that payload comes from the raw HTTP request viaTrafficDetector::getResult()— path info,SERVER_NAME, and theUser-Agent/Refererheaders. None were length-checked, but all five columns arevarchar(255)(methodisvarchar(10)).A scanner hitting a long URL, or any client sending an oversized
User-AgentorReferer, produced an insert PostgreSQL rejected. The message then burned its 3 retries and was dropped, so each occurrence silently lost a page view.The same hole existed in
ProcessDailyTrafficMessageHandler, which buildsPageViewfrom parsed log entries with no length guard either.Fix
PageViewnow enforces its own column widths in the constructor, so both producers are covered by one change:#[Orm\Column]attributes and the truncation — the mapping and the guard can't drift apart.mb_substr(..., 'UTF-8'). Code points are the unit PostgreSQL counts forcharacter varying, so a 255-code-point value always fits.s()->slice()was deliberately avoided:UnicodeStringslices on grapheme clusters, and an emoji-heavy user agent could still overflow the column after a 255-grapheme slice.refererkeeps itsnull.Why truncate rather than widen the columns
Request URLs have no upper bound, and
urlcarries a btree index (url_idx) — PostgreSQL caps index entries at ~2704 bytes, soTEXTwould just trade this error forindex row size exceeds maximum. Losing the tail of an outlier URL is the right trade against losing the page view.No migration needed —
doctrine:schema:update --dump-sqlreports the schema in sync, confirming the constants resolve identically inside the attributes.Tests
New
tests/Unit/Analytics/Domain/Entity/PageViewTest.phpcovers pass-through of values that fit, truncation of all six string fields, code-point-vs-byte cutting, and the null referer.Note
Messages that already failed were removed from the transport after exhausting their retries, so this prevents recurrence — it does not recover the page views already lost.