👨🏻💻 Session Highlight: "ClickHouse in Action: Building a Real-Time Ad Analytics Platform" by Siwakorn Ruenrit, Software Engineer at LINE MAN Wongnai.
Here are 6 key takeaways for anyone looking to scale their data architecture:
1. Calculating Unique Users (Reach) Requires "Raw Data" Unique customer metrics cannot be pre-aggregated because summing them up across multiple days leads to duplicated counts. For our use case, storing raw data and calculating it on the fly using an approximation function like uniq() is the simplest and most accurate approach.
2. The Right Tech Answers Your "Time-to-Market" We didn't choose ClickHouse because it's the "best tool in the world," but because it's easy to test (with cloud options for real-spec trials), has a growing local community for quick troubleshooting, and perfectly met our goal of launching the feature within a tight timeframe.
3. Custom Ingestion Requires "Careful Planning" and Batch Processing Custom data ingestion into ClickHouse requires careful planning. ClickHouse doesn't handle frequent, row-by-row inserts well, so writing custom code to pack data into large batches (10,000+ rows) is crucial. Furthermore, your rollout strategy matters heavily; our team used the Recreate strategy to prevent data duplication during system deployments.
4. If Updates Are Necessary... Choose the Right Table Engine Always remember that ClickHouse is an OLAP database—it wasn't built for frequent updates/deletes like a typical OLTP. If your use case mandates updates, opt for specialized table engines like ReplacingMergeTree, CollapsingMergeTree, or CoalescingMergeTree each handles data consolidation through background merges rather than in-place row updates.
5. Speed Up Unconventional Queries with "Projections" If you need to query data from a different angle than the main table's sorting key (ORDER BY), Projections are the answer. Acting similarly to a secondary index or a hidden materialized view, they speed up searches without the hassle of creating and maintaining an entirely new table.
6. Use Logical Clusters for Schema Consistency Across Nodes In a self-hosted ClickHouse setup, you can configure a logical cluster separate from the main data cluster. This is particularly useful for tables where you want identical data on every shard, such as a schema versions table for database migrations.
🎥 Want to explore system scaling and ClickHouse techniques?
Watch the full session playback right here! 👉 https://lnkd.in/g377pjys
#lifeatLMWN #LINEMANWongnai #TechTalks2026 #TechCommunity #TechMeetUp
#ClickHouse #DataEngineering #SystemDesign #DataArchitecture #SoftwareEngineering
