The massive increase of data and the existence of several distinct methods in which data is generated, accumulated, stored, and utilized has significantly changed with time. Additionally, the nature ...of data has changed throughout the years transforming from structured to more unstructured data. This brings about a need for efficient storage and management of such data which cannot be handled by traditional RDBMS methods. Hence, NoSQL databases have gained popularity and have become pivotal in database management. Finance is one domain where dynamic and large amounts of data is produced on a daily basis, thereby making NoSQL databases an ideal choice for data management. This paper compares these types of NoSQL databases based on certain metrics like data model, indexing methods, atomicity, integrity and several more and demonstrate implementation of three NoSQL databases namely, MongoDB, Cassandra and Redis, using financial data. Experiments were performed to compare the performance of the aforementioned databases when using fundamental READ queries to retrieve the complete dataset and complex READ queries to retrieve a specific section. Aggregation operations were also implemented on the data. Fundamental WRITE queries to load the entire dataset and complex WRITE queries to update particular parts of it were also performed.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
BackgroundThe comprehensive research facility for fusion technology magnet performance research platform (MPRP) is a large-scale experimental platform established for advanced superconducting magnet ...experiments. The retrieval speed of MPRP historical data is slow due to massive storage.PurposeThe study aims to develop a MPRP data archiving system (MPDAS) and increase its retrieval speed.MethodsFirst of all, the experimental physics and industrial control system (EPICS) data archiving plug-in was designed for MPDAS. Both MongoDB Sharding and Replica Set mechanism were employed to build a highly scalable data storage architecture. Then, the core ideas of three traditional cache replacement algorithms, LRU (least recently used), LFU (least frequently used) and FIFO (first in first out) were drawn by MPDAS to establish a data temperature model based on Newton's law of cooling. A multi-dimensional feature data partitioning algorithm was implemented to integrate access time, access frequency and storage order, henc
P315.69; 聚变堆主机关键系统的综合研究设施的磁体性能研究平台(Magnet Performance Research Platform,MPRP)是为先进超导磁体实验建立的大型实验平台,其历史数据在海量存储情况下存在检索速度慢的问题.因此,对系统检索速度进行研究并开发了MPRP数据归档系统(MPRP Data Archiving ...System,MPDAS).MPDAS设计了EPICS(Experimental Physics and Industrial Control System)数据归档插件并采用MongoDB分片和副本集机制搭建高扩展性数据存储架构.为提高数据检索速度,MPDAS借鉴最近最少使用(Least Recently Used,LRU)、使用频率最低(Least Frequently Used,LFU)、先进先出(First In First Out,FIFO)三种传统缓存替换算法核心思想,基于牛顿冷却定律建立数据温度模型并提出一种综合访问时间、访问频率以及存储顺序的多维度特征数据划分算法.根据数据划分算法标识冷热历史数据实现数据分层存储.MPDAS在查询历史数据时优先访问Redis,根据命中结果和数据完整性选择不同的检索策略.系统测试结果表明:MPDAS功能特征满足设计要求,其搭载的冷热数据划分算法相比FIFO、LRU、LFU在热数据库保存1％历史数据量时的Redis命中率分别提升了38.05％、26.91％和11.06％.通过提高热数据命中率能够直接减少数据检索平均响应时间,MPDAS通过量化历史数据热度并进行冷热划分,有效地提升了系统检索响应速度.
The rapid growth of data in all fields makes NoSQL database spectacular in the recent times. NoSQL gives the efficient storing of the huge volume of information in a data warehouse. The NoSQL server ...provides horizontal scalability and retrieval of data in the very prominent way. These features make the NoSQL database meaningfully increase in the real-time applications relating to the web services. Handling of extensive data in different areas makes sense of NoSQL. This work particularly uses the aviation data frame describing the operations being taking part in the aircraft systems such as productions and developments. The stable growth in aviation department leads to the global development. Hence risk is analyzed in all aspects. The branches of aviation are public aviation, general aviation, and defense aircraft keeps generating extensive records from the various third parties. These files which are in Gigabytes (GB) and Terabytes (TB) are stored in the databases for many practical applications. The data generated in various aviation departments are handled using document store NoSQL. The aviation framework using document store enables the accessing and storing information in the NoSQL database server efficiently with recovering the deleted data from the MongoDB data file structure. This work focuses on using aggregation operation to retrieve the data in an efficient manner and Mongo index B-tree algorithm where insertion, deletion, sequential access, and searches are done in logarithmic time
Значна кількість сучасних розробників використовують платформу .NET для створення програм, що працюють із базами даних. Cosmos DB стає все більш популярним вибором як NoSQL-сховище для баз даних. ...Cosmos DB – гнучка й масштабована система, і правильний вибір відповідного АРІ в програмній реалізації може значно вплинути на продуктивність самих програм. Cosmos DB надає різні API для роботи з усіма типами баз даних. Кожен із цих API може бути використаний за допомогою різних методів програмної реалізації. Предметом дослідження є програмні реалізації на платформі .NET під різні Cosmos DB API. Під час обрання найбільш підхожого Cosmos DB API на платформі .NET розробникам може допомогти не тільки документація, але й результати експериментальних досліджень АРІ, що дасть змогу покращити якість коду й продуктивність самих систем. Мета роботи – підвищити ефективність програмних розробок на платформі .NET, що використовують Cosmos DB API, шляхом створення рекомендацій щодо обрання методів програмної реалізації API на основі результатів експериментального дослідження. Завдання статті: дослідити та порівняти методи програмної реалізації Cosmos DB API шляхом вивчення продуктивності різних типів запитів на цих програмних рішеннях; проаналізувати здобуті результати та розробити рекомендації з використання методів. Методи: багатокритеріальний аналіз Cosmos DB API, логічне моделювання даних, дослідження. Результати: розроблено програмні рішення на основі використання CosmosClient, Entity Framework Core для Cosmos DB API for NoSQL та на основі MongoClient для Cosmos DB API for MongoDB; проведено серію експериментів і вимірювань показників продуктивності для кожного з програмних рішень; проаналізовано здобуті результати та запропоновано рекомендації з використання розглянутих методів програмної реалізацій Cosmos DB API на платформі .NET. Висновки. Загалом вибір програмного підходу залежить від конкретного завдання, але дослідження показали, що Cosmos DB API for NoSQL із застосуванням CosmosClient – це найкращий вибір для незначних проєктів, а з використанням Entity Framework Core Cosmos підходить для проєктів з більшими обсягами інформації та складними запитами. Якщо в проєкті застосовується MongoDB, то відповідне рішення з використанням MongoClient є кращим варіантом, ніж Cosmos DB API for NoSQL.
NoSQL database management systems are very diverse and are known to evolve very fast. With so many NoSQL database options available nowadays, it is getting harder to make the right choice for certain ...use cases. Also, even for a given NoSQL database management system, performance may vary significantly between versions. Database performance benchmarking shows the actual performance for different scenarios on different hardware configurations in a straightforward and precise manner. This paper presents a NoSQL database performance study in which two of the most popular NoSQL database management systems (MongoDB and Apache Cassandra) are compared, and the analyzed metric is throughput. Results show that Apache Cassandra outperformes MongoDB in an update heavy scenario only when the number of operations is high. Also, for a read intensive scenario, Apache Cassandra outperformes MongoDB only when both number of operations and degree of parallelism are high.
Marine big data are characterized by a large amount and complex structures, which bring great challenges to data management and retrieval. Based on the GeoSOT Grid Code and the composite index ...structure of the MongoDB database, this paper proposes a spatio-temporal grid index model (STGI) for efficient optimized query of marine big data. A spatio-temporal secondary index is created on the spatial code and time code columns to build a composite index in the MongoDB database used for the storage of massive marine data. Multiple comparative experiments demonstrate that the retrieval efficiency adopting the STGI approach is increased by more than two to three times compared with other index models. Through theoretical analysis and experimental verification, the conclusion could be achieved that the STGI model is quite suitable for retrieving large-scale spatial data with low time frequency, such as marine big data.
Currently, humans live in an era of data oceans, where the amount of data production is increasing from time to time, which is followed by severe challenges in terms of processing, storing, and ...analyzing data, especially big data. The increase in the number of large data production can affect the speed of access to the database, effectiveness, and speed of response time in the data processing. Relational databases have been the leading model for data storage, analysis, processing, and retrieval for more than forty years. However, due to the increasing need for large-scale data storage, the scalability and performance of a data processing system, as well as the constant growth of the amount of data, another alternative to databases emerged, namely NoSQL technology. Based on previous studies regarding the comparison of response time and database performance, the average concludes that NoSQL performance is more effective and efficient than relational databases. Based on the implementation and testing, it can be concluded that the NoSQL database application MongoDB is proven to be superior in every command of CRUD tested compared to the Elasticsearch NoSQL database application, where in testing the create data command with a JSON file, the MongoDB database application is 42.5 times faster than the Elasticsearch database application. In testing the command to create data into a database containing different amounts of data, the MongoDB database application is 333.9 times faster than the average response time of the Elasticsearch database application. In testing the read command for data in a database containing different amounts of data, the MongoDB database application is 35.5 times faster than the Elasticsearch database application. In testing the update operation of data in a database containing different amounts of data, the MongoDB database application is 9.8 times faster than the Elasticsearch database application. in testing the delete operation of data in a database containing different amounts of data, the MongoDB database application is 58.9 times faster than the Elasticsearch database application.
PDF
