Akademska digitalna zbirka SLovenije - logo
E-viri
Recenzirano Odprti dostop
  • Debugging large-scale data ...
    Rezig, El Kindi; Brahmaroutu, Ashrita; Tatbul, Nesime; Ouzzani, Mourad; Tang, Nan; Mattson, Timothy; Madden, Samuel; Stonebraker, Michael

    Proceedings of the VLDB Endowment, 08/2020, Letnik: 13, Številka: 12
    Journal Article

    Data pipelines are the new code. Consequently, data scientists need new tools to support the often time-consuming process of debugging their pipelines. We introduce Dagger , an end-to-end system to debug and mitigate data-centric errors in data pipelines, such as a data transformation gone wrong or a classifier underperforming due to noisy training data. Dagger supports inter-module debugging, where the pipeline blocks are treated as black boxes, as well as intra-module debugging, where users can debug data objects in Python scripts (e.g., DataFrames). In this demo, we will walk the audience through a rich, real-world business intelligence use case from our industrial collaborators at Intel, to highlight how Dagger enables data scientists to productively identify and mitigate data-centric problems at different stages of pipeline development.