Akademska digitalna zbirka SLovenije - logo
E-viri
Recenzirano Odprti dostop
  • Ignis: An efficient and sca...
    Piñeiro, César; Martínez-Castaño, Rodrigo; Pichel, Juan C.

    Future generation computer systems, April 2020, 2020-04-00, Letnik: 105
    Journal Article

    Most of the relevant Big Data processing frameworks (e.g., Apache Hadoop, Apache Spark) only support JVM (Java Virtual Machine) languages by default. In order to support non-JVM languages, subprocesses are created and connected to the framework using system pipes. With this technique, the impossibility of managing the data at thread level arises together with an important loss in the performance. To address this problem we introduce Ignis, a new Big Data framework that benefits from an elegant way to create multi-language executors managed through an RPC system. As a consequence, the new system is able to execute natively applications implemented using non-JVM languages. In addition, Ignis allows users to combine in the same application the benefits of implementing each computational task in the best suited programming language without additional overhead. The system runs completely inside Docker containers, isolating the execution environment from the physical machine. A comparison with Apache Spark shows the advantages of our proposal in terms of performance and scalability. •A new step forward toward the real convergence of HPC and Big Data worlds.•Efficient execution of multi-language applications without additional overhead.•Outperforms Spark considering some of the most typical algorithmic models.•Ignis API inspired by Spark to facilitate the adoption by the research community.•A completely isolated framework running inside Docker containers.