Reinforcement R-learning Model for Time Scheduling of On-demand Fog Placement

• Post by: Hani Sami
• 25 June 2021
• Comments off

Abstract

On the fly deployment of fog nodes near users provides the flexibility of pushing services anywhere and whenever needed. Nevertheless, taking a real-life scenario, the cloud might limit the number of fogs to place for minimizing the complexity of monitoring a large number of fogs and cost for volunteers that do not offer their resources for free. This implies choosing the right time and best volunteer to create a fog which the cloud can benefit from is essential. This choice is subject to study the demand of a particular location for services in order to maximize the resources utilization of these fogs. A simple algorithm will not be able to explore randomly changing users’ demands. Therefore, there is a need for an intelligent model capable of scheduling fog placement based on the user’s requests. In this paper, we propose a \textbf{Fog Scheduling Decision} model based on reinforcement R-Learning, which focuses on studying the behavior of service requesters and produces a suitable fog placement schedule based on the concept of average reward. Our model aims to decrease the cloud’s load by utilizing the maximum available fogs resources over different locations. An implementation of our proposed R-learning model is provided in the paper, followed by a series of experiments on a real dataset to prove its efficiency in utilizing fog resources and minimizing the cloud’s load. We also demonstrate the ability of our model to improve over time by adapting the new demand of users. Experiments comparing the decisions of our model with two other potential fog placement approaches used for task/service scheduling (THRESHOLD-based and RANDOM-based) show that the number of processed requests performed by the cloud decreases from 100\% to 30\% with a limited number of fogs to push. These results demonstrate that our proposed Fog Scheduling Decision model plays a crucial role in the placement of the on-demand fog to the right location at the right time while taking into account the user’s needs.

Authors’ Notes

Pushing on-demand fogs to volunteering devices everywhere has its disadvantages also. Here we mention the potential high complexity of monitoring all fog placements \cite{fatema2014survey}, which the cloud is responsible for. Moreover, Having to push services everywhere can be considered as a security threat for the service provider, especially when services get controller by volunteers \cite{roman2018mobile}. In addition, volunteers will, at some point, ask for a reward for their offered services, which the cloud should pay \cite{yi2015survey}. All of these factors lead to the need for minimizing the number of fog placement. Therefore these are considered as the main motivations behind limiting the number of fog placement by the cloud. This now raises another challenge on the cloud, which is making fog placement decisions at the time and place that best maximizes its profit by maximizing the utilization of fog resources. Simple approaches are not enough for the cloud to make good scheduling decisions for cloud placement. However, there should be a model that studies users’ demands to decide on the best time and place to schedule the fog. Pushing fogs to places having high demands of services can significantly improve the decision efficiency by maximizing the fog resources utilization and therefore minimizing the load on the cloud by having to deal with fewer requests. An example of possible approaches for fog scheduling are the THRESHOLD-based and RANDOM-based approaches which we prove in the paper that they are not applicable in real life.

In this paper, we address the aforementioned problem by proposing a Fog Scheduling Decision model based on a reinforcement learning technique called R-Learning. R-learning has a similar concept to the Q-learning technique. They both use Q-tables, states, actions, reward, and loss. However, the main difference is that R-learning uses the concept of average reward and not the discounted reward \cite{gosavi2015simulation}. In other words, our problem does not converge to a static end goal, but rather a general goal of minimizing the cloud load. This is done by maximizing the total average reward of transitioning between all states and taking actions. R-learning is an area of reinforcement learning that aims to maximize the total reward by taking a certain action on a given state and performing punishment whenever the model fails to meet the expectation\cite{gosavi2015simulation} \cite{watkins1992q}. In this context, our proposed approach predicts the time and location of the needed fogs that mostly minimize the number of requests processed by the cloud while serving the biggest number of requests generated by users or IoT devices and therefore maximize the usage of volunteering resources. The predictions of this algorithm help increasing the QoS as well as maintaining a low pressure on the cloud resources. This is done through studying user’s behavior of making requests to services hosted on the cloud. Users’ requests can be tracked using the server’s logs. A real dataset is used from the logs of the Nasa server \cite{nasaDataSet}, and experiments are conducted on the implemented scheduling model to prove its efficiency and enhancements achieved over time by taking near-optimal service scheduling decisions. We also compare our proposed model to the decisions taken by the THRESHOLD and RANDOM-based approaches to show the achieved improvement.\
The rest of this paper is organized as follows. In section 2, we present some background information and study the current work surrounding the use of the time scheduling model and the effect of its absence. In section 3, we introduce an overview of the architecture and methodology used where our proposed model plays the main role. In section 4, we propose the Fog Scheduling Decision model and explain the mathematical formulas and R-Learning algorithm behind it. Section 5 is dedicated to analyzing the results generated by our experiments. Finally, we conclude the paper and present future directions in section 6.