Massive Data Processing in Adobe Using Delta Lake

At Adobe Experience Platform, we ingest TBs of data every day and manage PBs of data for our customers as part of the Unified Profile Offering.

At the heart of this is a bunch of complex ingestion of a mix of normalized and denormalized data with various linkage scenarios power by a central Identity Linking Graph. This helps power various marketing scenarios that are activated in multiple platforms and channels like email, advertisements etc. We will go over how we built a cost-effective and scalable data pipeline using Apache Spark and Delta Lake and share our experiences.

Speaker: Yeshwanth Vijayakumar – Adobe