In the ETL process the transformation of data is achieved through the execution of a set of transformation operations. The realization of this process (the order in which the transformation ...operations must be executed) should be preceded by a specification of the transformation process at a higher level of abstraction. The specification is given through mappings representing abstract operations specific to the transformation process. These mappings are defined through weaving models and metamodels. A generated weaving metamodel (GWMM) is proposed giving the complete mapping semantics through specific link types (representing the abstract operations) and appropriate OCL constraints. Weaving models specifying the actual mappings must be in accordance with this proposed GWMM.
Les besoins fonctionnels et non fonctionnels représentent la première brique pour la conception de toute application, logiciel, système, etc. L'ensemble des traitements associés aux besoins est ...établi dans le cadre de l'ingénierie des Besoins (IB). Le processus de l'IB comporte plusieurs étapes consistant à découvrir, analyser, valider et faire évoluer l'ensemble des besoins relatifs aux fonctionnalités du système. La maturité de la communauté de l'IB lui a permis d'établir un cycle de vie bien déterminé pour le processus de besoins qui comprend les phases suivantes :l'élicitation, la modélisation, la spécification, la validation et la gestion des besoins. Une fois ces besoins validés, ils sont archivés ou stockés dans des référentiels ou des dépôts au sein des entreprises. Avec l'archivage continu des besoins, ces entreprises disposent d'une mine d'informations qu'il faudra analyser afin de reproduire les expériences cumulées et le savoir-faire acquis en réutilisant et en exploitant ces besoins pour des nouveaux projets. Proposer à ces entreprises un entrepôt dans lequel l'ensemble de besoins est stocké représente une excellente opportunité pour les analyser à des fins décisionnelles et les fouiller pour reproduire des anciennes expériences. Récemment, la communauté des processus (BPM) a émis le même besoin pour les processus. Dans cette thèse, nous souhaitons exploiter le succès des entrepôts de données pour le reproduire sur les besoins fonctionnels. Les problèmes rencontrés lors de la conception des entrepôts de données se retrouvent presque à l'identique dans le cas des besoins fonctionnels.Ces derniers sont souvent hétérogènes, surtout dans le cas d'entreprises de grande taille comme Airbus, où chaque partenaire a la liberté d'utiliser ses propres vocabulaire et formalisme pour décrire ses besoins. Pour réduire cette hétérogénéité, l'appel aux ontologies est nécessaire. Afin d'assurer l'autonomie de chaque source, nous supposons que chaque source a sa propre ontologie.Cela nécessite des efforts de matching entre les ontologies afin d'assurer l' intégration des besoins fonctionnels. Une particularité importante liée à l'entreposage de besoins réside dans le fait que ces derniers sont souvent exprimés à l'aide des formalismes semi-formels comme les use cases d'UML avec une partie textuelle importante. Afin de nous rapprocher le plus possible de ce que nous avons fait dans le cadre de l'entreposage de données, nous proposons un modèle pivot permettant de factoriser trois semi-formalismes répandus utilisés par les sources de besoins avec une description précise de ces derniers. Ce modèle pivot permettra de définir le modèle multidimensionnel del' entrepôt de besoins, qui sera ensuite alimenté par les besoins des sources en utilisant un algorithme ETL (Extract, Transform, Load). À l'aide des mécanismes de raisonnement offerts par les ontologies et des métriques de matching, nous avons nettoyé notre entrepôt de besoins. Une fois l'entrepôt déployé, il est exploité par des outils d'analyse OLAP.Notre méthodologie est supportée par un outil couvrant l'ensemble des phases de conception et d'exploitation d'un entrepôt de besoins.
Functiona and non-functional requirements represent the first step for the design of any application, software, system, etc. Ail the issues associated to requirements are analyzed in the Requirements Engineering (RE) field. The RE process consists of several steps consisting of discovering, analyzing, validating and evolving the requirements related to the functionalities of the system. The RE community proposed a well-defined life-cycle for the requirements process that includes the following phases: elicitation, modeling, specification, validation and management. Once the requirements are validated, they are archived or stored in repositories in companies. With the continuous storage of requirements, companies accumulate an important amount of requirements information that needs to be analyzed in order to reproduce the previous experiences and the know-how acquired by reusing and exploiting these requirements for new projects. Proposing to these companies a warehouse in which all requirements are stored represents an excellent opportunity to analyze them for decision-making purposes. Recently, the Business Process Management Community (BPM) emitted the same needs for processes. In this thesis, we want to exploit the success of data warehouses and to replicate it for functional requirements. The issues encountered in the design of data warehouses are almost identical in the case of functional requirements. Requirements are often heterogeneous, especially in the case of large companies such Airbus, where each panner bas the freedom to use its own vocabulary and formalism to describe the requirements. To reduce this heterogeneity, using ontologies is necessary. In order to ensure the autonomy of each partner, we assume that each source bas its own ontology. This requires matching efforts between ontologies to ensure the integration of functional requirements. An important feature related to the storage of requirements is that they are often expressed using semi-forma! formalisms such as use cases of UML with an important textual part. In order to get as close as possible to our contributions in data warehousing,we proposed a pivot model factorizing three well-known semi-formalisms. This pivot model is used to define the multidimensional model of the requirements warehouse, which is then alimented by the sources requirements using an ETL algorithm (Extract,Transform, Load).Using reasoning mechanisms otfered by ontologies and matching metrics, we cleaned up our requirements warehouse. Once the warehouse is deployed, it is exploited using OLAP analysis tools. Our methodology is supported by a tool covering all design phases of the requirements warehouse
This paper points out the possibilities of using MATLAB in processing a large amount of measured data. An application created calculates and then graphically displays changes in individual ...statistical parameters of the measured mains voltage. Although the application works with voltage values, it can also be used for similar evaluation of changes in statistical parameters of other measured technical quantities, where the measured data are collected, which will be subsequently evaluated.
Data Migration using ETL Workflow Saranya, N; Brindha, R; Aishwariya, N ...
2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS),
2021-March-19, Volume:
1
Conference Proceeding
Data Migration is a method of transferring data from one device to another. It is often a sub-activity of the implementation of a business application. From a theoretical stand point, we investigate ...data migration fundamentals in this paper. To increase the efficiency of static analysis, we also implement methods like data validation, ETL process, Migration of data using Talend and also cloud computing.
BACKGROUND: A data warehouse (DW) is an integrated collection of subject-oriented data in the support of decision making. Importantly, the integration of data sources is achieved through the use of ...ETL (Extract, Transform, and Load) processes. It is therefore extensively recognized that the appropriate design of the ETL processes are key factors in the success of DW projects. OBJECTIVE: We assess existing research proposals about ETL process modeling for data warehouse in order to identify their main characteristics, notation, and activities. We also study if these modeling approaches are supported by some kind of prototype or tool. METHOD: We have undertaken a systematic mapping study of the research literature about modeling ETL processes. A mapping study provides a systematic and objective procedure for identifying the nature and extent of the available research by means of research questions. RESULTS: The study is based on a comprehensive set of papers obtained after using a multi-stage selection criteria and are published in international workshops, conferences and journals between 2000 and 2009. CONCLUSIONS: This systematic mapping study states that there is a clear classification of ETL process modeling approaches, but that they are not enough covered by researchers. Therefore, more effort is required to bridge the research gap in modeling ETL processes.
Ad-hoc OLAP on Trajectory Data Marketos, Gerasimos; Theodoridis, Yannis
2010 Eleventh International Conference on Mobile Data Management
Conference Proceeding
The application of Data Warehousing (DW) and OLAP techniques on conventional data has been extensively studied in the literature. On the other hand, Trajectory Data Warehousing and Trajectory OLAP ...are relatively new research areas, which have to deal with the spatiotemporal (hence dynamic) nature of such data. In this paper, we present an innovative organization of a trajectory data cube in order to be able to answer OLAP queries considering different interpretations of the notion of trajectory. Thus, ad-hoc analysis on trajectory data cubes can be achieved, which can be really useful for a number of applications. Preliminary experimental results illustrate the applicability and efficiency of our approach.
This paper focuses on integrating food data sources into a central database using extract, transform and load processing and the subsequent data quality enhancement. The obtained data will be ...transmitted by a food data web service to certain health apps for further use. Furthermore, it is planned to identify inconsistent, incorrect, duplicate and incomplete data using methods of data profiling so that they can be corrected. In order to quantify the data quality purposefully and appropriately, certain quality metrics were used. These metrics were calculated and evaluated using random test data selected from the food data.
Data warehouse (DW) design is based on a set of requirements expressed as service level agreements (SLAs) and business level objects (BLOs). Populating a DW system from a set of information sources ...is realized with extract-transform-load (ETL) processes based on SLAs and BLOs. The entire task is complex, time consuming, and hard to be performed manually. This paper presents our approach to the requirement-driven creation of ETL designs. Each requirement is considered separately and a respective ETL design is produced. We propose an incremental method for consolidating these individual designs and creating an ETL design that satisfies all given requirements. Finally, the design produced is sent to an ETL engine for execution. We illustrate our approach through an example based on TPC-H and report on our experimental findings that show the effectiveness and quality of our approach.
This article presents the development of an application (App) for mobile devices to visualize hotspots in the road network of Madrid metropolitan area, which is also aimed at warning drivers when ...approaching those hotspots. The paper describes, firstly, the nature of data used and their provenance, and then puts the focus on the Extraction, Transformation and Loading Process (or ETL Process) carried out for the generation of the structured data used by the App. Afterwards, the main features and functionalities of the developed App are also described.
PDF
