Dynamic resource allocation for large-scale streaming data

Wenyi Chen, Purdue University

Abstract

The data from network cameras can be valuable for a wide range of scientific studies such as weather, wildlife, and traffic. The resource demands may fluctuate substantially for some of these studies (for example, seasonal or during only rush hours). Cloud computing's pay-peruse can be a preferred solution for analyzing large amounts of data from these network cameras. There is no study focusing on how to allocate cloud resources analyzing many video streams from network cameras. Existing auto-scaling and load balancing are inapplicable due to the nature of the workloads. We presents the first solution for managing cloud resources analyzing multiple video streams. When an analysis program is launched, the resource manager adjusts the resource allocation by one of two methods: gradually scaling up the number of instances or predicting the needed number of instances. Then, the resource manager monitors the utilization and scales the resources in response to the analysis programs loads. The two different ways have their pros and cons. Gradually scaling up is guaranteed to reach a good solution, but may take many trials. Prediction runs in constant time, but the prediction may be incorrect thus needs correction. The proposed resource manager is implemented using Microsoft Azure and demonstrated that the system can adaptively adjust resource to cater to the workload. We experiment the system with different sizes of virtual machine (VM) and found that smaller size usually works better in performance and cost.

Degree

M.S.E.C.E.

Advisors

Lu, Purdue University.

Subject Area

Computer Engineering

COinS