Holistic data provision at Conrad
Consistent and secure data provision in multichannel e-commerce – Conrad’s cloud-based ETL processes
Conrad Electronic markets products primarily from the area of electronics and technology. The company attempts to meet the full range of its customers’ technical operating needs by means of their digital procurement platform. This encompasses more than 6 million product offerings from over 4,500 manufacturers, from individual spare parts to high quantities. Processing Conrad’s own approximately 700,000 items in full batches already used to take up to 24 hours. That timespan was already far too long for important marketing activities and the requirement of being able to respond immediately to legal issues in the shop.
The additional goal of launching a new marketplace with over 7 million items on the platform would also entail a tenfold increase in the existing data processing operations. At the time, Conrad lacked the appropriate product data processes for processing the enormous data load (over 500 gigabytes) involved in publishing new products and updates of product information in PIM or ERP as quickly as possible, securely, and in a standardized manner over all channels.
Conrad was using Talend’s enterprise solution for data integration on the e-commerce platform. However, the technical complexity was mapped in more than 100 ETL job logics. Ensuring continuity in the data supply required Conrad’s Data Services team to be on call 24/7 and to engage in intensive maintenance activities. After a detailed analysis of the current situation, Conrad’s Data Services team, together with the ETL Developers team from Avantgarde Labs, decided to make 4 sustainable modifications in Conrad’s product data logistics:
Despite a tenfold increase in the data volume, the time required to process the product data was reduced by 85%. The solution was based on a delta processing logic in Talend and the horizontal scaling provided within the Google Cloud platform. Dedicated monitoring of the processes now allowed prioritized responses to errors to ensure a stable supply of data for the shop, marketplace and external channels.
Delta processing combined with horizontal scaling in the Google Cloud enables secure and fully automated provision of more than 7 million items of product data for the shop, marketplace and external users, with a new item update rate of only up to one hour. Hardware resources are accessed only when needed. The average utilization of the Google Compute Engines is at least 50 percent, which delivers high cost efficiency with the hardware used. The temporary scaling brought about stable operating times in the supply of data. In the event that overloading nonetheless occurs, this can be recognized early on through the enhanced monitoring. The Data Services team is able to intervene in dependence on the priority and thereby safeguard ongoing operations. By providing many customized views of the product data, the Data Services team reduces the amount of time-consuming data processing required by other consumers. The reuse of data saves resources enterprise-wide and ensures data consistency across vertical team boundaries.
"Avantgarde Labs has provided invaluable help in making our data processes faster, more scalable, and more robust to meet the challenges of the rapidly growing data volume and increased demand for up-to-date data. They gave us support with the planning and implementation as well as in operation. This has put Conrad in a position to be able to continue providing the customers on our platform the best possible services."