Skip to search boxSkip to navigationSkip to main content

Real-time user clickstream behavior analysis based on Apache Storm streaming

  • Gautam Pal
    ,
  • Katie Atkinson
    ,
  • Gangmin Li
  • University of Liverpool
Research Output: Contribution to journal Article Peer-review

Open access

Abstract

This paper presents an approach to analyzing consumers’ e-commerce site usage and browsing motifs through pattern mining and surfing behavior. User-generated clickstream is first stored in a client site browser. We build an ingestion pipeline to capture the high-velocity data stream from a client-side browser through Apache Storm, Kafka, and Cassandra. Given the consumer’s usage pattern, we uncover the user’s browsing intent through n-grams and Collocation methods. An innovative clustering technique is constructed through the Expectation-Maximization algorithm with Gaussian Mixture Model. We discuss a framework for predicting a user’s clicks based on the past click sequences through higher order Markov Chains. We developed our model on top of a big data Lambda Architecture which combines high throughput Hadoop batch setup with low latency real-time framework over a large distributed cluster. Based on this approach, we developed an experimental setup for an optimized Storm topology and enhanced Cassandra database latency to achieve real-time responses. The theoretical claims are corroborated with several evaluations in Microsoft Azure HDInsight Apache Storm deployment and in the Datastax distribution of Cassandra. The paper demonstrates that the proposed techniques help user experience optimization, building recently viewed products list, market-driven analyses, and allocation of website resources.

Publication Information

Output type

Research Output: Contribution to journal Article Peer-review

Original language

English

Pages from-to (Number of pages)

Pages 1829-1859 (31 pages)

Journal (Volume, Issue Number)

Electronic Commerce Research (Volume 23, Issue 3)

Publication milestones

  • Accepted/In press - 16/10/2021
  • Published - 22/12/2021

Publication status

Published - 22/12/2021

ISSN

1389-5753

External Publication IDs

  • handle.net: 10547/625365
  • Scopus: 85121532441

Access to documents

Publication metrics

Metrics

Download statistics
Download count
2