{"id":8872,"date":"2022-07-22T14:59:56","date_gmt":"2022-07-22T12:59:56","guid":{"rendered":"https:\/\/avantgarde-labs.com\/case-studies\/holistic-data-provision-at-conrad\/"},"modified":"2022-07-22T16:00:27","modified_gmt":"2022-07-22T14:00:27","slug":"holistic-data-provision-at-conrad","status":"publish","type":"case_studies","link":"https:\/\/avantgarde-labs.com\/en\/case-studies\/holistic-data-provision-at-conrad\/","title":{"rendered":"Holistic data provision at Conrad"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

The Challenge<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tConrad Electronic markets products primarily from the area of electronics and technology. The company attempts to meet the full range of its customers\u2019 technical operating needs by means of their digital procurement platform. This encompasses more than 6 million product offerings from over 4,500 manufacturers, from individual spare parts to high quantities.<\/span>\r\n\r\nProcessing Conrad\u2019s own approximately 700,000 items<\/strong> in full batches already used to take up to 24 hours<\/strong>. That timespan was already far too long for important marketing activities and the requirement of being able to respond immediately to legal issues in the shop.\u00a0<\/span>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe additional goal of launching a new marketplace with over 7 million items on the platform would also entail a <\/span>tenfold increase in the existing<\/b> data processing operations.\u00a0<\/span>\r\n\r\nAt the time<\/span>, Conrad lacked<\/b> the appropriate <\/span>product data processes<\/b> for <\/span>processing the enormous data load<\/b> (over 500 gigabytes) involved in publishing new products and updates of product information in PIM or ERP as quickly as possible, securely, and in a standardized manner over all channels.\u00a0<\/span>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

The Avantgarde Labs solution<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tConrad was using Talend’s enterprise solution for data integration on the e-commerce platform. However, the technical complexity was mapped in more than 100 ETL job logics. Ensuring continuity in the data supply required Conrad’s Data Services team to be on call 24\/7 and to engage in intensive maintenance activities.<\/span>\r\n\r\nAfter a detailed analysis of the current situation, Conrad’s Data Services team, together with the ETL Developers team from Avantgarde Labs, decided to make 4 sustainable modifications in Conrad’s product data logistics:<\/strong><\/span>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t
  • \n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\tTransform the previous batch processes into a delta processing logic<\/span>\n\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t
  • \n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\tIntroduce a horizontal scaling strategy in the cloud<\/span>\n\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t<\/ul>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t
      \n\t\t\t\t\t\t\t
    • \n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\tDevelop an internal data hub<\/span>\n\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t
    • \n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\tExpand the monitoring and introduce automatic maintenance services<\/span>\n\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t<\/ul>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\tDespite a <\/span>tenfold increase in the data volume<\/b>, the time required to process the product data <\/span>was reduced by 85%<\/b>. The solution was based on a delta processing logic<\/strong> in Talend and the horizontal scaling provided within the Google Cloud platform. Dedicated monitoring<\/strong> of the processes now allowed prioritized responses to errors to ensure a stable supply of data for the shop, marketplace and external channels.<\/span>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t

      Project results & customer benefits<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\tDelta processing combined with horizontal scaling in the Google Cloud enables <\/span>secure and fully automated provision<\/b> of more than 7 million items of product data for the shop, marketplace and external users, with a <\/span>new item update rate<\/b> of only up\u00a0to one hour<\/b>.<\/span>\r\n\r\nHardware resources are accessed only when needed. The average utilization of the Google Compute Engines is at least 50 percent, which delivers <\/span>high cost efficiency<\/b> with the hardware used.\u00a0<\/span>\r\n\r\nThe temporary scaling brought about <\/span>stable operating times<\/b> in the supply of data. In the event that overloading nonetheless occurs, this can be recognized <\/span>early on<\/b> through the enhanced monitoring. The Data Services team is able to intervene in dependence on the priority and thereby safeguard ongoing operations.\u00a0<\/span>\r\n\r\nBy providing many customized views of the product data, the Data Services team reduces the amount of time-consuming data processing required by other consumers. The reuse of data <\/span>saves resources enterprise-wide<\/b> and ensures <\/span>data consistency<\/b> across vertical team boundaries.<\/span>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t
      \n\t\t\t\t\t\t\t
      \"Avantgarde Labs has provided invaluable help in making our data processes faster, more scalable, and more robust to meet the challenges of the rapidly growing data volume and increased demand for up-to-date data. They gave us support with the planning and implementation as well as in operation. This has put Conrad in a position to be able to continue providing the customers on our platform the best possible services.\"<\/div>\n\t\t\t\n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t\t\t
      \n\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t
      Markus Wiesinger<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
      Senior Expert Data Service Engineer
      Conrad Electronic SE<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

      Consistent and secure data provision in multichannel e-commerce \u2013 Conrad\u2019s cloud-based ETL processes<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"template":"","meta":{"_acf_changed":false,"site-sidebar-layout":"no-sidebar","site-content-layout":"page-builder","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"disabled","ast-breadcrumbs-content":"","ast-featured-img":"disabled","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"branche":[226],"solutions":[275,274],"cooperations":[245],"services":[247,248],"teams":[276],"class_list":["post-8872","case_studies","type-case_studies","status-publish","hentry","branche-trade","solutions-e-commerce-data-management","solutions-multichannel-integration-en","cooperations-time-materials","services-application-development","services-data-management","teams-etl-developers"],"acf":[],"_links":{"self":[{"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/case_studies\/8872","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/case_studies"}],"about":[{"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/types\/case_studies"}],"author":[{"embeddable":true,"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/users\/1"}],"version-history":[{"count":3,"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/case_studies\/8872\/revisions"}],"predecessor-version":[{"id":8915,"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/case_studies\/8872\/revisions\/8915"}],"wp:attachment":[{"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/media?parent=8872"}],"wp:term":[{"taxonomy":"branche","embeddable":true,"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/branche?post=8872"},{"taxonomy":"solutions","embeddable":true,"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/solutions?post=8872"},{"taxonomy":"cooperations","embeddable":true,"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/cooperations?post=8872"},{"taxonomy":"services","embeddable":true,"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/services?post=8872"},{"taxonomy":"teams","embeddable":true,"href":"https:\/\/avantgarde-labs.com\/en\/wp-json\/wp\/v2\/teams?post=8872"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}