This study focuses on solving the vehicle routing problem (VRP) of E-logistics service providers. In our problem, each vehicle must visit some pick up nodes first, for instance, warehouses to pick up the orders then makes deliveries for customers in the list. Each pickup node has its own list of more than one customers requiring delivery. The objective is to minimize the total travelling cost while real-world application constraints, such as heterogeneous vehicles, capacity limits, time window, driver working duration, etc. are still considered. This research firstly proposes a mathematical model for this multiple pickup and multiple delivery vehicle routing problem with time window and heterogeneous fleets (MPMDVRPTWHF). In the next step, the ant colony optimization algorithm is studied to solve the problem in the large-scale.