{"id":1786,"date":"2023-08-31T23:13:14","date_gmt":"2023-08-31T23:13:14","guid":{"rendered":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/?page_id=1786"},"modified":"2023-09-01T20:28:08","modified_gmt":"2023-09-01T20:28:08","slug":"rl-real-time-scheduling","status":"publish","type":"page","link":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/research\/rl-real-time-scheduling\/","title":{"rendered":"RL-real-time-scheduling"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-page\" data-elementor-id=\"1786\" class=\"elementor elementor-1786\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-e45912d elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"e45912d\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-505f290\" data-id=\"505f290\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-044077a elementor-widget__width-initial hfe-nav-menu__align-left hfe-submenu-icon-arrow hfe-submenu-animation-none hfe-link-redirect-child hfe-nav-menu__breakpoint-tablet elementor-widget elementor-widget-navigation-menu\" data-id=\"044077a\" data-element_type=\"widget\" data-e-type=\"widget\" data-settings=\"{&quot;padding_horizontal_menu_item&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:15,&quot;sizes&quot;:[]},&quot;padding_horizontal_menu_item_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;padding_horizontal_menu_item_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;padding_vertical_menu_item&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:15,&quot;sizes&quot;:[]},&quot;padding_vertical_menu_item_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;padding_vertical_menu_item_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;menu_space_between&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;menu_space_between_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;menu_space_between_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;menu_row_space&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;menu_row_space_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;menu_row_space_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;dropdown_border_radius&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;top&quot;:&quot;&quot;,&quot;right&quot;:&quot;&quot;,&quot;bottom&quot;:&quot;&quot;,&quot;left&quot;:&quot;&quot;,&quot;isLinked&quot;:true},&quot;dropdown_border_radius_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;top&quot;:&quot;&quot;,&quot;right&quot;:&quot;&quot;,&quot;bottom&quot;:&quot;&quot;,&quot;left&quot;:&quot;&quot;,&quot;isLinked&quot;:true},&quot;dropdown_border_radius_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;top&quot;:&quot;&quot;,&quot;right&quot;:&quot;&quot;,&quot;bottom&quot;:&quot;&quot;,&quot;left&quot;:&quot;&quot;,&quot;isLinked&quot;:true},&quot;width_dropdown_item&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;220&quot;,&quot;sizes&quot;:[]},&quot;width_dropdown_item_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;width_dropdown_item_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;padding_horizontal_dropdown_item&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;padding_horizontal_dropdown_item_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;padding_horizontal_dropdown_item_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;padding_vertical_dropdown_item&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:15,&quot;sizes&quot;:[]},&quot;padding_vertical_dropdown_item_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;padding_vertical_dropdown_item_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;distance_from_menu&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;distance_from_menu_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;distance_from_menu_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;toggle_size&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;toggle_size_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;toggle_size_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;toggle_border_width&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;toggle_border_width_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;toggle_border_width_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;toggle_border_radius&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;toggle_border_radius_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;toggle_border_radius_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]}}\" data-widget_type=\"navigation-menu.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t<div class=\"hfe-nav-menu hfe-layout-horizontal hfe-nav-menu-layout horizontal hfe-pointer__none\" data-layout=\"horizontal\">\n\t\t\t\t<div role=\"button\" class=\"hfe-nav-menu__toggle elementor-clickable\" tabindex=\"0\" aria-label=\"Menu Toggle\">\n\t\t\t\t\t<span class=\"screen-reader-text\">Menu<\/span>\n\t\t\t\t\t<div class=\"hfe-nav-menu-icon\">\n\t\t\t\t\t\t<i aria-hidden=\"true\"  class=\"fas fa-align-justify\"><\/i>\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<nav class=\"hfe-nav-menu__layout-horizontal hfe-nav-menu__submenu-arrow\" data-toggle-icon=\"&lt;i aria-hidden=&quot;true&quot; tabindex=&quot;0&quot; class=&quot;fas fa-align-justify&quot;&gt;&lt;\/i&gt;\" data-close-icon=\"&lt;i aria-hidden=&quot;true&quot; tabindex=&quot;0&quot; class=&quot;far fa-window-close&quot;&gt;&lt;\/i&gt;\" data-full-width=\"yes\">\n\t\t\t\t\t<ul id=\"menu-1-044077a\" class=\"hfe-nav-menu\"><li id=\"menu-item-1815\" class=\"menu-item menu-item-type-post_type menu-item-object-page parent hfe-creative-menu\"><a href=\"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/research\/rl-real-time-scheduling\/\" class = \"hfe-menu-item\">RL-real-time-scheduling<\/a><\/li>\n<li id=\"menu-item-1799\" class=\"menu-item menu-item-type-post_type menu-item-object-page parent hfe-creative-menu\"><a href=\"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/research\/rl-real-time-scheduling\/synopsis\/\" class = \"hfe-menu-item\">Synopsis<\/a><\/li>\n<li id=\"menu-item-1814\" class=\"menu-item menu-item-type-post_type menu-item-object-page parent hfe-creative-menu\"><a href=\"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/research\/rl-real-time-scheduling\/team-members\/\" class = \"hfe-menu-item\">Team Members<\/a><\/li>\n<li id=\"menu-item-1812\" class=\"menu-item menu-item-type-post_type menu-item-object-page parent hfe-creative-menu\"><a href=\"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/research\/rl-real-time-scheduling\/educational-activities\/\" class = \"hfe-menu-item\">Educational Activities<\/a><\/li>\n<li id=\"menu-item-1811\" class=\"menu-item menu-item-type-post_type menu-item-object-page parent hfe-creative-menu\"><a href=\"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/research\/rl-real-time-scheduling\/outreach-and-broader-impact\/\" class = \"hfe-menu-item\">Outreach and Broader Impact<\/a><\/li>\n<li id=\"menu-item-1813\" class=\"menu-item menu-item-type-post_type menu-item-object-page parent hfe-creative-menu\"><a href=\"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/research\/rl-real-time-scheduling\/publications-code\/\" class = \"hfe-menu-item\">Publications &amp; Code<\/a><\/li>\n<\/ul> \n\t\t\t\t<\/nav>\n\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-top-column elementor-element elementor-element-23c0af3\" data-id=\"23c0af3\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-f007c32 elementor-widget elementor-widget-text-editor\" data-id=\"f007c32\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>NSF project 2203239: CNS Core: Small: Reinforcement Learning for Real-time Wireless Scheduling and Edge Caching: Theory and Algorithm Design.<\/p><p><strong>Overview:<\/strong><\/p><p><span style=\"color: var( --e-global-color-text );font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, 'Noto Sans', sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol', 'Noto Color Emoji';font-size: var( --e-global-typography-primary-font-size );font-weight: var( --e-global-typography-primary-font-weight )\">5G and Beyond (B5G) technology promises to offer enhanced mobile broadband\u00a0 and ultra-reliable low-latency communications\u00a0 services. Indeed, <\/span>recent years have witnessed a tremendous growth in real-time\u00a0 applications\u00a0 in wirelessly networked systems, such as connected cars and multi-user augmented reality (AR).\u00a0 \u00a0For instance, for connected cars,\u00a0 coordinated sensing and mobility control rely heavily on\u00a0 real-time information exchange among vehicles. Wireless edge caching is\u00a0 \u00a0another emerging B5G application requiring high bandwidth, where optimal caching decisions would depend on the cache contents and dynamic user demand profiles. A census is that conventional approaches, using model-based optimization to tackle the challenges in guaranteeing ultra-low latency and high bandwidth services,\u00a0 may not work well\u00a0 in some complicated B5G settings, calling for machine learning based solutions.<\/p><p><strong class=\"h4Style02\">Technical approaches and main tasks:<\/strong><\/p><p>\u00a0 In this project, recent advances on offline reinforcement learning\u00a0 (RL) will be leveraged to study\u00a0 two important problems in B5G, namely\u00a0 1) deadline-aware wireless scheduling to guarantee low latency and 2) edge caching to achieve\u00a0 high bandwidth content delivery, as outlined below:<\/p><p><strong>Task 1:<\/strong> Deadline-aware scheduling of real-time traffic has been a long-standing open problem, despite significant effort using model-based optimization. In Task 1,\u00a0 deadline-aware scheduling policies will be trained using physics-aided offline RL, ready to be used for online scheduling. Specifically,\u00a0 \u00a0the Actor-Critic (A-C) method will be used for offline training of scheduling policies, consisting of two phases: 1)\u00a0initialization of Actor structure via behavioral cloning and 2)\u00a0 policy improvement via the physics-aided A-C method. The underlying\u00a0 rationale is as follows:\u00a0 \u00a0With a good model-based scheduling algorithm as the initial actor structure,\u00a0 \u00a0the A-C method can be leveraged to yield a better scheduling policy, thanks to its nature of policy improvement.\u00a0Further,\u00a0 innovative algorithms will be devised to address the outstanding problems in the A-C method, namely overestimation bias and high variance, and Meta-RL will be used for adaptation to distribution shift in nonstationary network dynamics.\u00a0 \u00a0The\u00a0 \u00a0PIs have taken initial steps and obtained promising\u00a0 \u00a0results, and will pursue\u00a0 in-depth investigation.<\/p><p><strong>Task 2: <\/strong>The focus of Task 2 is on wireless edge caching,\u00a0 an application where the storage capacities\u00a0 at both the network edge and user devices are harnessed to alleviate the need of high-bandwidth communications over long distances. The combinatorial nature of joint communication and caching optimization herein, with the uncertainties of\u00a0 system dynamics, calls for non-trivial design of machine learning algorithms.\u00a0 We will leverage deep RL to investigate\u00a0 wireless edge caching. The preliminary\u00a0 study by the PIs, inherently respecting system constraints, has yield interesting initial results.\u00a0 The PIs will continue along the lines of the preliminary work to generalize the framework to study coded caching.\u00a0<\/p><div>\u00a0<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Menu RL-real-time-scheduling Synopsis Team Members Educational Activities Outreach and Broader Impact Publications &amp; Code NSF project 2203239: CNS Core: Small: Reinforcement Learning for Real-time Wireless Scheduling and Edge Caching: Theory and Algorithm Design. Overview: 5G and Beyond (B5G) technology promises to offer enhanced mobile broadband\u00a0 and ultra-reliable low-latency communications\u00a0 services. Indeed, recent years have witnessed [&hellip;]<\/p>\n","protected":false},"author":64,"featured_media":0,"parent":35,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"elementor_header_footer","meta":{"inline_featured_image":false,"footnotes":""},"class_list":["post-1786","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/wp-json\/wp\/v2\/pages\/1786","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/wp-json\/wp\/v2\/users\/64"}],"replies":[{"embeddable":true,"href":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/wp-json\/wp\/v2\/comments?post=1786"}],"version-history":[{"count":20,"href":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/wp-json\/wp\/v2\/pages\/1786\/revisions"}],"predecessor-version":[{"id":1930,"href":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/wp-json\/wp\/v2\/pages\/1786\/revisions\/1930"}],"up":[{"embeddable":true,"href":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/wp-json\/wp\/v2\/pages\/35"}],"wp:attachment":[{"href":"https:\/\/faculty.engineering.ucdavis.edu\/jzhang\/wp-json\/wp\/v2\/media?parent=1786"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}