MapReduce/Hadoop framework has been widely used to process large-scale datasets on computing clusters. Scheduling map tasks with data locality consideration is crucial to the performance of ...MapReduce. Many works have been devoted to increasing data locality for better efficiency. However, to the best of our knowledge, fundamental limits of MapReduce computing clusters with data locality, including the capacity region and theoretical bounds on the delay performance, have not been well studied. In this paper, we address these problems from a stochastic network perspective. Our focus is to strike the right balance between data locality and load balancing to simultaneously maximize throughput and minimize delay. We present a new queueing architecture and propose a map task scheduling algorithm constituted by the Join the Shortest Queue policy together with the MaxWeight policy. We identify an outer bound on the capacity region, and then prove that the proposed algorithm can stabilize any arrival rate vector strictly within this outer bound. It shows that the outer bound coincides with the actual capacity region, and the proposed algorithm is throughput-optimal. Furthermore, we study the number of backlogged tasks under the proposed algorithm, which is directly related to the delay performance based on Little's law. We prove that the proposed algorithm is heavy-traffic optimal, i.e., it asymptotically minimizes the number of backlogged tasks as the arrival rate vector approaches the boundary of the capacity region. Therefore, the proposed algorithm is also delay-optimal in the heavy-traffic regime. The proofs in this paper deal with random processing times with heterogeneous parameters and nonpreemptive task execution, which differentiate our work from many existing works on MaxWeight-type algorithms, so the proof techniques themselves for the stability analysis and the heavy-traffic analysis are also novel contributions.
Network Traffic Monitoring and Analysis (NTMA) represents a key component for network management, especially to guarantee the correct operation of large-scale networks such as the Internet. As the ...complexity of Internet services and the volume of traffic continue to increase, it becomes difficult to design scalable NTMA applications. Applications such as traffic classification and policing require real-time and scalable approaches. Anomaly detection and security mechanisms require to quickly identify and react to unpredictable events while processing millions of heterogeneous events. At last, the system has to collect, store, and process massive sets of historical data for post-mortem analysis. Those are precisely the challenges faced by general big data approaches: Volume, Velocity, Variety, and Veracity. This survey brings together NTMA and big data. We catalog previous work on NTMA that adopt big data approaches to understand to what extent the potential of big data is being explored in NTMA. This survey mainly focuses on approaches and technologies to manage the big NTMA data, additionally briefly discussing big data analytics (e.g., machine learning) for the sake of NTMA. Finally, we provide guidelines for future work, discussing lessons learned, and research directions.
Abstract
Since the last two decades a lot of efforts were made in order to produce methodologies there are able to model traffic characteristics and present anticipative traffic condition. New ...software programs include a lot of parameters that can affect how a simulation approaches reality. In order to minimize the resources needed for the analysis of a traffic scheme, it is necessary to use a method that allows the modification of only the critical parameters. In this study, the main factors with major influence in the calibration and validation of microsimulation models were identified. Based on predetermined graphics, users have the ability to complete faster a traffic study.
The Internet of Things comprises wireless devices with limited computing resources. It targets attacks that exploit vulnerabilities such as unencrypted data transfer. Conventional vulnerability ...detection occurs from databases that list the most common vulnerabilities and exploits (CVEs). However, these bases are limited to known vulnerabilities, which is not the case for the IoT context most of the time. This work proposes MANDRAKE: a Method for vulnerAbilities detectioN baseD on the IoT netwoRk pAcKEt traffic using machine learning techniques. A performance evaluation has been conducted in a smart home scenario taking as basis two datasets, one generated experimentally for this work and the other from the literature. The results have achieved 99% precision in detecting vulnerabilities in network traffic.
Abstract
Traffic flow detection plays a significant part in freeway traffic surveillance systems. Currently, effective autonomous traffic analysis is a challenging task due to the complexity of ...traffic delays, despite the significant investment spent by authorities in monitoring and analysing traffic congestion. This study builds an intelligent analytic method based on machine‐learning algorithms to investigate and predict road traffic flows in four locations in the United Kingdom (London, Yorkshire and the Humber, North East, and North West) with a range of relevant factors. While aiming to conduct the study, the dataset ‘estimated annual average daily flows (AADFs) Data—major and minor roads’ from the UK government was used. Machine‐learning algorithms are used for this research and classification applied consists of Logistic Regression, Decision Trees, Random Forests, K‐Nearest Neighbors, and Gradient Boosting. Each of these algorithms achieves an accuracy of over 93% and the F1 score of over 95%, with Random Forest outperforming the other algorithms. This analytical approach helps to focus attention on critical areas to reduce traffic flows on major and minor roads in the area. In summary, the findings on traffic analysis have been discussed in detail to demonstrate the practical insights of this study.
Tor is the most popular anonymization network with millions of daily users. This makes it an attractive target for attacks, e.g., by malicious autonomous systems (ASs) performing active routing ...attacks to become man in the middle and deanonymize users. It was shown that the number of such malicious ASs is significantly larger than previously expected due to the lack of security guarantees in the Border Gateway Protocol (BGP). In response, recent works suggest alternative Tor path selection methods preferring Tor nodes with higher resilience to active BGP attacks.
In this work, we analyze the implications of such proposals and demonstrate that two state-of-the-art path selection methods, namely Counter-RAPTOR and DPSelect, are not as secure as thought before. First, we show that DPSelect achieves only one third of its originally claimed resilience and, thus, is not as resilient as widely accepted. Second, we reveal that the resilience to active BGP attacks on the way back, i.e., from the first anonymization node to a given Tor user, provided by both methods is significantly lower than on the forward path. Beside their lower resilience in specific cases, we show that for particular users the usage of Counter-RAPTOR and DPSelect also leads to leakage of user’s location. Furthermore, we uncover the performance implications of both methods and identify scenarios where their usage leads to significant performance bottlenecks. Finally, we propose a new metric to quantify the user’s location leakage by path selection. Using this metric and performing large-scale analysis, we show to which extent a malicious Tor middle node can fingerprint the user’s location and the confidence it can achieve. Our findings shed light on the implications of path selection methods on the users’ anonymity and the need for further research.
The prosperity of the Internet has made network traffic analysis increasingly indispensable in network operation. With the development of machine learning, more researchers and engineers are using ...deep learning models for network traffic analysis. However, the rapidly growing data size and model complexity make resource scheduling a serious limitation, which is why cloud computing services are typically required for network analysis. To leverage the advantages of serverless platforms, we propose faaShark , an end-to-end network traffic analysis system based on a serverless computing platform. faaShark adapts distributed training to fully utilize the flexibility of serverless platforms. Additionally, we design a cold start optimization algorithm to reduce the hit rate of cold start when serving pretrained models to handle network analysis requests. Our extensive experiments evaluate the impact of several parameters of distributed training and confirm the effectiveness of our cold start optimization algorithm when building such a network analysis system atop serverless computing frameworks.
Ransomware is a type of advanced malware that can encrypt a user’s files or lock a computer system until a ransom has been paid. Ransomworm is a type of malware that combines the payload of ...ransomware with the propagation feature of a computer worm. Most host-based detection methods require the host to be infected and the payload to be executed first to be able to identify anomalies and detect the malware. By the time of infection, it might too late as some of the system’s assets would have been already encrypted or exfiltrated by the malware. On the contrary, the network-based methods can be one of the crucial means in detecting ransomworm activities when it attempts to spread to infect other networks before executing the payload. Therefore, a thorough analysis of ransomworm network traffic can be one of the essential means for early detection. This paper presents a comprehensive behavioral analysis of ransomworm network traffic, taking WannaCry, which launched a worldwide cyberattack, and NotPetya as a case study. Two sets of related features were extracted based on two independent flow levels: session-based and time-based. On top of each set, an independent classifier was built. Moreover, to improve the reliability, a multi-feature and multi-classifier network-based system, MFMCNS, has been proposed. MFMCNS employs these classifiers working in parallel on different flow levels, then it adopts a fusion rule to combine the classifiers’ decisions. The experimental results prove that MFMCNS is reliable and has high detection accuracy.
Abstract
The research presented was on the table more years and covers the cooperation of the University staff with City of Bratislava binding with the activity of private company. The base task was ...how to change the tremendous destruction of modal split in Bratislava, which growing on the side of private car usage. Results of couple of traffic studies show, that how could be this change re-define the current traffic behaviour and create for the new area new functional interest and certainly the new quality of relationships of land use. The system solution is in the amplification by a capacitive new tramway network. In the conditions of Bratislava, we need to change the behaviour of the people in the mobility over the area of the Capital city of Slovakia. The task was to solve the problem in Bratislava agglomeration as well. The solution is shown as a part of the integrated public transport services possibility to operate regular all days’ mobility problems in the city and not only to cover the rush hours. This article is an overview of 5 years work of traffic engineers and planners to keep clear systematic approach and solutions with developers and municipality administration.
PDF
