In power market studies, the forecast of locational marginal price (LMP) relies on the load forecasting results from the viewpoint of planning. It is well known that short-term load forecasting results always carry certain degree of errors mainly due to the random nature of the load. At the same time, LMP step changes occur at critical load levels (CLLs). Therefore, it is interesting to investigate the impact of load forecasting uncertainty on LMP. With the assumption of a certain probability distribution of the actual load, this paper proposes the concept of probabilistic LMP and formulates the probability mass function of this random variable. The expected value of probabilistic LMP is then derived, as well as the lower and upper bound of its sensitivity. In addition, two useful curves, alignment probability of deterministic LMP versus forecasted load and expected value of probabilistic LMP versus forecasted load, are presented. The first curve is designed to identify the probability that the forecasted price in a deterministic LMP matches the actual price at the forecasted load level. The second curve is demonstrated to be smooth and therefore eliminates the step changes in deterministic LMP forecasting. This helps planners avoid the possible sharp changes during decision-making process. The proposed concept and method are illustrated with a modified PJM five-bus system and the IEEE 118-bus system.