Add page view tracking for admin analytics #20
Conversation
This proposal outlines a comprehensive page view tracking system that enables administrators to: - Track hot routes, endpoints, and UI hotspots with minimal overhead - Analyze user behavior and navigation patterns in real-time - Monitor API endpoint performance and usage - Generate actionable insights with sub-millisecond tracking overhead Key features: - Real-time analytics dashboard with live updates - Click heatmap visualization for UI interaction analysis - API endpoint performance monitoring - Privacy-focused design (GDPR compliant) - Scalable architecture using Redis streams and BullMQ Technical stack: Next.js middleware, Redis streams, BullMQ, tRPC, PostgreSQL with time-series partitioning, React with D3.js visualization Target overhead: < 0.5ms per page view event
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const results = await db | ||
| .select({ | ||
| route: pageViews.route, | ||
| views: sql<number>`count(*)`, | ||
| uniqueVisitors: sql<number>`count(distinct ${pageViews.sessionId})`, | ||
| avgTimeOnPage: sql<number>`avg(extract(epoch from (lead(${pageViews.timestamp}) over (partition by ${pageViews.sessionId} order by ${pageViews.timestamp}) - ${pageViews.timestamp})))` | ||
| }) |
There was a problem hiding this comment.
The getHotRoutes query nests a lead(...) over (...) window function inside avg(...) while also grouping by route. Postgres does not allow window functions inside aggregate expressions in the same select list, so this SQL will fail to compile when the resolver runs. To compute average time on page you need a subquery that calculates the per-row lead difference first and then aggregates it.
Useful? React with 👍 / 👎.
| // Create consumer group if not exists | ||
| redis.xgroup('CREATE', STREAM_KEY, CONSUMER_GROUP, '0', 'MKSTREAM').catch(() => {}); | ||
|
|
||
| // Worker to continuously consume from stream | ||
| async function processEventStream() { | ||
| while (true) { | ||
| try { | ||
| // Read batch of events | ||
| const results = await redis.xreadgroup( | ||
| 'GROUP', CONSUMER_GROUP, CONSUMER_NAME, | ||
| 'BLOCK', 5000, // 5 second timeout | ||
| 'COUNT', BATCH_SIZE, | ||
| 'STREAMS', STREAM_KEY, '>' | ||
| ); | ||
|
|
||
| if (!results || results.length === 0) continue; | ||
|
|
||
| const [streamName, messages] = results[0]; | ||
|
|
||
| // Process batch | ||
| await processBatch(messages); | ||
|
|
||
| // Acknowledge messages | ||
| const messageIds = messages.map(([id]) => id); | ||
| await redis.xack(STREAM_KEY, CONSUMER_GROUP, ...messageIds); |
There was a problem hiding this comment.
The event processor continuously reads with XREADGROUP ... STREAMS tracking:events '>' and immediately acknowledges messages on success, but it never inspects the consumer group’s pending entry list or reclaims unacknowledged records after failures or restarts. If the worker dies after reading a batch but before xack, those events remain pending forever and are never reprocessed because subsequent reads with '>' skip them. This design risks losing tracking data any time a worker crashes or the process is redeployed.
Useful? React with 👍 / 👎.
Enhanced the proposal to integrate Temporal workflows for reliable, long-running analytics operations: Key additions: - Temporal workflows for scheduled aggregation (1min, 1hour, 1day rollups) - Data retention workflow with automatic cleanup and partition management - Report generation workflow for scheduled analytics reports - Clear separation of concerns: BullMQ for high-throughput event processing, Temporal for long-running, reliable scheduled tasks Architecture improvements: - Added detailed Temporal workflow and activity code examples - Workflow scheduling with cron expressions - Overlap policies to prevent concurrent executions - Workflow visibility via Temporal UI for monitoring Benefits: - Leverages existing Temporal infrastructure in corestack - Built-in retry and error handling for critical aggregation jobs - Durable workflow state survives system restarts - Consistent with the rest of the corestack architecture - Cron scheduling built-in (no need for external schedulers) Updated implementation phases to include Temporal workflow development in Phase 3.
This proposal outlines a comprehensive page view tracking system that enables administrators to:
Key features:
Technical stack: Next.js middleware, Redis streams, BullMQ, tRPC, PostgreSQL with time-series partitioning, React with D3.js visualization
Target overhead: < 0.5ms per page view event