Real-time clickstream analytics engine

Challenge

Ekstra Bladet needed to analyse vast volumes of live clickstream data from its front page — structured, queryable with sub-minute latency for complex analytical workloads, and retained for a month before archiving.

Solution

An auto-scaling, serverless ingestion and analytics pipeline on AWS, built with infrastructure-as-code.

• Streaming ETL. Events captured via an application load balancer into Kinesis Firehose, processed by Lambda functions.
• Low-latency warehouse. Data lands in Redshift for PostgreSQL-compatible analytical SQL, with raw and processed data archived in S3.
• Live visualisation. An R Shiny dashboard shows current clickstream patterns to business-intelligence consumers.

Results

• Vast event volumes ingested in real time
• Serverless processing at scale
• Sub-minute latency for analytical SQL
• Live dashboard deployed