The usage of live streaming services has led to a substantial increase in live video traffic. However, the perceived quality of experience of users is frequently limited by variations in the upstream ...bandwidth of streamers. To address this issue, several adaptive bitrate (ABR) algorithms have been developed to mitigate bandwidth variations. Nevertheless, the ability of users to enjoy high-quality live streams remains limited. While neural-enhanced approaches, such as super-resolution, offer significant quality improvements, frame-oriented super-resolution leads to excessive inference delay that violates the real-time feature of live streaming. In response, we propose ViChaser, which examines block-oriented super-resolution for live streaming. ViChaser performs neural super-resolution on potential blocks of interest in the media server, corresponding to the user's viewpoint, and uses online learning to adapt to the dynamic content of the video. Additionally, ViChaser utilizes the Lyapunov framework to efficiently allocate uplink bandwidth for original low-quality live video and high-quality labels. The experimental results demonstrate that ViChaser achieves 1.2-1.5 dB higher video quality in Peak-Signal-to-Noise-Ratio than WebRTC and increases processing speed by 11-16 fps relative to LiveNAS.
Bandwidth prediction is critical in any Real-time Communication (RTC) service or application. This component decides how much media data can be sent in real time. Subsequently, the video and audio ...encoder dynamically adapts the bitrate to achieve the best quality without congesting the network and causing packets to be lost or delayed. To date, several RTC services have deployed the heuristic-based Google Congestion Control (GCC), which performs well under certain circumstances and falls short in some others. In this paper, we leverage the advancements in reinforcement learning and propose BoB (Bang-on-Bandwidth) - a hybrid bandwidth predictor for RTC. At the beginning of the RTC session, BoB uses a heuristic-based approach. It then switches to a learning-based approach. BoB predicts the available bandwidth accurately and improves bandwidth utilization under diverse network conditions compared to the two winning solutions of the ACM MMSys'21 grand challenge on bandwidth estimation in RTC. An open-source implementation of BoB is publicly available for further testing and research.
A robust and language agnostic Voice Activity Detection (VAD) is crucial for Digital Entertainment Content (DEC). Primary examples of DEC include movies and TV series. Some ways in which VAD systems ...are used for DEC creation include augmenting subtitle creation, subtitle drift detection and correction, and audio diarisation. Majority of the previous work on VAD focuses on scenarios that: (a) have minimal background noise, and (b) where the audio content is delivered in English language. However, movies and TV shows can: (a) have substantial amounts of non-voice background signal (e.g. musical score and environmental sounds), and (b) are released worldwide in a variety of languages. This makes most of the previous standard VAD approaches not readily applicable for DEC related applications. Furthermore, there does not exist a comprehensive analysis of Deep Neural Network’s (DNN) performance for the task of VAD applied to DEC. In this work, we present a thorough survey on DNN based VADs on DEC data in terms of their accuracy, Area Under Curve (AUC), noise sensitivity, and language agnostic behaviour. For our analysis we use 1100 proprietary DEC videos spanning 450 h of content in 9 languages and 5 + genres, making our study the largest of its kind ever published. The key findings of our analysis are: (a) even high quality timed-text or subtitle 22subtitles and timed-text are used interchangeably in the manuscript files contain significant levels of label-noise (up to 15%). Despite high label noise, deep networks are robust and are able to retain high AUCs (∼ 0.94). (b) Using larger labelled dataset can substantially increase neural VAD model’s True Positive Rate (TPR) with up to 1.3% and 18% relative improvement over current state-of-the-art methods in Hebbar et al. (2019) and Chaudhuri et al. (2018) respectively. This effect is more pronounced in noisy environments such as music and environmental sounds. This insight is particularly instructive while prioritizing domain specific labelled data acquisition versus exploring model structure and complexity. (c) Currently available sequence based neural models show similar levels of competence in terms of their language agnostic behaviour for VAD at high Signal-to-Noise Ratios (SNRs) and for clean speech, (d) Deep models exhibit varied performance across different SNRs with CLDNN (Zazo et al., 2016) being the most robust, and (e) models with comparatively larger number of parameters (∼2 M) are less robust to input noise as opposed to models having smaller number of parameters (∼ 0.5 M).
Congestion Control for Web Real-Time Communication Carlucci, Gaetano; De Cicco, Luca; Holmer, Stefan ...
IEEE/ACM transactions on networking,
2017-Oct., 2017-10-00, 20171001, Letnik:
25, Številka:
5
Journal Article
Recenzirano
Applications requiring real-time communication (RTC) between Internet peers are ever increasing. RTC requires not only congestion control but also minimization of queuing delays to provide ...interactivity. It is known that the well-established transmission control protocol congestion control is not suitable for RTC due to its retransmissions and in-order delivery mechanisms, which induce significant latency. In this paper, we propose a novel congestion control algorithm for RTC, which is based on the main idea of estimating-using a Kalman Filter-the end-to-end one-way delay variation which is experienced by packets traveling from a sender to a destination. This estimate is compared with a dynamic threshold and drives the dynamics of a controller located at the receiver, which aims at maintaining queuing delays low, while a loss-based controller located at the sender acts when losses are detected. The proposed congestion control algorithm has been adopted by Google Chrome. Extensive experimental evaluations have shown that the algorithm contains queuing delays while providing intra and inter protocol fairness along with full link utilization.
Online video streaming has fundamental limitations on the transmission bandwidth and computational capacity and super-resolution is a promising potential solution. However, applying existing video ...super-resolution methods to online streaming is non-trivial. Existing video codecs and streaming protocols (e.g., WebRTC) dynamically change the video quality both spatially and temporally, which leads to diverse and dynamic degradations. Furthermore, online streaming has a strict requirement for latency that most existing methods are less applicable. As a result, this paper focuses on the rarely exploited problem setting of online streaming video super resolution. To facilitate the research on this problem, a new benchmark dataset named LDV-WebRTC is constructed based on a real-world online streaming system. Leveraging the new benchmark dataset, we propose a novel method specifically for online video streaming, which contains a convolution and Look-Up Table (LUT) hybrid model to achieve better performance-latency trade-off. To tackle the changing degradations, we propose a mixture-of-expert-LUT module, where a set of LUT specialized in different degradations are built and adaptively combined to handle different degradations. Experiments show our method achieves 720P video SR around 100 FPS, while significantly outperforms existing LUT-based methods and offers competitive performance compared to efficient CNN-based methods. Code is available at https://github.com/quzefan/ConvLUT .
As chronic diseases and demographic changes alter the trends in world population, more pressure is put on health management to deal efficiently with a new scenario of longer life expectancy and ...chronic disabilities. Advances in telecommunication technology and miniaturization of sensors in combination with medical information technology provide a viable solution to reduce healthcare costs and deliver remote medical services through connected devices. Although remote consultation via video-conferencing has been well established, many of the chronic or long-term musculoskeletal conditions require pro-active management and therapy. There is a need to develop more advanced interactive telerehabilitation systems that support real-time remote delivery of physical therapy sessions into patients’ homes. In this paper, we introduce KinectRTC, an innovative framework that can be used for Kinect-based telerehabilitation with efficient real-time transmission of video, audio, depth, and skeletal data. By taking advantage of the Web Real-Time Communication (WebRTC) technology, the proposed framework is able to manage video and audio streams based on the state of the network and the available bandwidth to guarantee the real-time performance of the communication.
•A real time communication framework based on Kinect and WebRTC is proposed.•It allows to stream audio, video, depth frames and arbitrary binary data.•It adapts data streaming to the state of the network ensuring real-time communication.•Tests show that the streaming is stable and reliable in different situations.
The attractiveness of real-time multimedia communication as part of an e-learning platform largely depends on the quality of the telecommunications infrastructure and on the services that support the ...exchange of audiovisual data. The research subject of this work is communication between stationary and mobile devices using distributed services such as Web Real-Time Communication (WebRTC) running in HTML5 compliant web browsers. The test connections were carried out in a peer-to-peer architecture over a local wireless WiFi network and a mobile network supporting the Long Term Evolution (LTE) standard for data interchange. Several audiovisual sessions between two clients were analyzed for different connection scenarios. Parameters responsible for the transmission quality, such as delay, jitter, packet loss, or the speed of sending and receiving video frames were measured for each scenario. Open audiovisual communication system performance experiments were conducted under real operating conditions. The obtained results indicated potential applications in the development of e-learning websites.
Full-body avatars are suggested to be beneficial for communication in virtual environments, and consistency between users' voices and gestures is considered essential to ensure communication quality. ...This paper propose extending the functionality of a web-based VR platform to support the use of full-body avatars and delegated avatar transforms synchronization to WebRTC DataChannel to enhance the consistency between voices and gestures. Finally, we conducted a preliminary validation to confirm the consistency.
Shared media experiences between geographically distributed users are gaining momentum. When remote users are consuming online media content together while interacting, several challenges still need ...to be addressed to provide an enjoyable experience (e.g. in terms of delays and synchronization). This paper presents Wersync, a web-based platform that enables the creation of different groups of geographically distributed users for concurrently consuming the same media content in a synchronized manner, while socially interacting via text and/or audiovisual chat channels, and even collaborating. Its main features have been derived via a thorough state-of-the-art analysis, results of a previous survey and user-centric activities. Interestingly, Wersync includes adaptive and accurate media synchronization solutions, which performance has been objectively evaluated, obtaining better results than the ones needed for the kind of applications Wersync is intended for, such as Social TV. Additionally, the features of the platform have been subjectively evaluated (N = 60 participants) in a social viewing scenario, obtaining satisfactory results, mainly in terms of perception about performance, originality, usability, attractiveness, usefulness, applicability and awakened interest. These results and the development based on standard web components foresee the potential impact and widespread adoption of the platform in the future.
PDF
