•Model both picking and replenishment activities in robotic warehouses.•Model synchronization of both pods and robots using nested fork-join queues.•Model dynamic resource reallocation of both robots and workstations in warehouses.•Capture time varying demand using Markov-modulated Poisson process.•Both warehouse R/W ratio and the peak demand affect resource allocation decision. A Robotic Mobile Fulfillment System (RMFS) is an automated parts-to-picker material handling system, in which robots carry pods with products to the order pickers. It is particularly suitable for e-commerce order fulfillment and can quickly and frequently reallocate workers and robots across the picking and replenishment processes to respond to strong demand fluctuations. More resources for the picking process means lower customer wait times, whereas more resources for the replenishment process means a higher inventory level and product availability. This paper models the RMFS as a queuing network and integrates it within a Markov decision process (MDP), that aims to allocate robots across the pick and replenishment processes during both high and low demand periods, based on the workloads in these processes. We extend existing MDP models with one resource type and one process to an MDP model for two resources types and two processes. The policies derived from the model are compared with benchmark policies from practice. The results show that the length of the peak demand phase and the height of the peak affects the optimal policy choice. In addition, policies that continually reallocate resources based on the workload outperform benchmark policies from practice. Moreover, if the number of robots is limited, continual resource reallocation can reduce costs sharply. The results show that optimal dynamic policies can reduce the cost by up to 52.18% on average compared to optimal fixed policies.