The Challenge
Conrad Electronic markets products primarily from the area of electronics and technology. The company attempts to meet the full range of its customers’ technical operating needs by means of their digital procurement platform. This encompasses more than 6 million product offerings from over 4,500 manufacturers, from individual spare parts to high quantities. Processing Conrad’s own approximately 700,000 items in full batches already used to take up to 24 hours. That timespan was already far too long for important marketing activities and the requirement of being able to respond immediately to legal issues in the shop.
The additional goal of launching a new marketplace with over 7 million items on the platform would also entail a tenfold increase in the existing data processing operations. At the time, Conrad lacked the appropriate product data processes for processing the enormous data load (over 500 gigabytes) involved in publishing new products and updates of product information in PIM or ERP as quickly as possible, securely, and in a standardized manner over all channels.
The Avantgarde Labs solution
Conrad was using Talend’s enterprise solution for data integration on the e-commerce platform. However, the technical complexity was mapped in more than 100 ETL job logics. Ensuring continuity in the data supply required Conrad’s Data Services team to be on call 24/7 and to engage in intensive maintenance activities. After a detailed analysis of the current situation, Conrad’s Data Services team, together with the ETL Developers team from Avantgarde Labs, decided to make 4 sustainable modifications in Conrad’s product data logistics:
- Transform the previous batch processes into a delta processing logic
- Introduce a horizontal scaling strategy in the cloud
- Develop an internal data hub
- Expand the monitoring and introduce automatic maintenance services
Despite a tenfold increase in the data volume, the time required to process the product data was reduced by 85%. The solution was based on a delta processing logic in Talend and the horizontal scaling provided within the Google Cloud platform. Dedicated monitoring of the processes now allowed prioritized responses to errors to ensure a stable supply of data for the shop, marketplace and external channels.
Project results & customer benefits
Delta processing combined with horizontal scaling in the Google Cloud enables secure and fully automated provision of more than 7 million items of product data for the shop, marketplace and external users, with a new item update rate of only up to one hour. Hardware resources are accessed only when needed. The average utilization of the Google Compute Engines is at least 50 percent, which delivers high cost efficiency with the hardware used. The temporary scaling brought about stable operating times in the supply of data. In the event that overloading nonetheless occurs, this can be recognized early on through the enhanced monitoring. The Data Services team is able to intervene in dependence on the priority and thereby safeguard ongoing operations. By providing many customized views of the product data, the Data Services team reduces the amount of time-consuming data processing required by other consumers. The reuse of data saves resources enterprise-wide and ensures data consistency across vertical team boundaries.
"Avantgarde Labs has provided invaluable help in making our data processes faster, more scalable, and more robust to meet the challenges of the rapidly growing data volume and increased demand for up-to-date data. They gave us support with the planning and implementation as well as in operation. This has put Conrad in a position to be able to continue providing the customers on our platform the best possible services."
Markus Wiesinger
Senior Expert Data Service Engineer
Conrad Electronic SE
Conrad Electronic SE
