The development of rice (Oryza sativa) seed is closely associated with assimilates storage and plant yield, and is fine controlled by complex regulatory networks. Exhaustive transcriptome analysis of developing rice embryo and endosperm will help to characterize the genes possibly involved in the regulation of seed development and provide clues of yield and quality improvement. Our analysis showed that genes involved in metabolism regulation, hormone response and cellular organization processes are predominantly expressed during rice development. Interestingly, 191 transcription factor (TF)-encoding genes are predominantly expressed in seed and 59 TFs are regulated during seed development, some of which are homologs of seed-specific TFs or regulators of Arabidopsis seed development. Gene co-expression network analysis showed these TFs associated with multiple cellular and metabolism pathways, indicating a complex regulation of rice seed development. Further, by employing a cold-resistant cultivar Hanfeng (HF), genome-wide analyses of seed transcriptome at normal and low temperature reveal that rice seed is sensitive to low temperature at early stage and many genes associated with seed development are down-regulated by low temperature, indicating that the delayed development of rice seed by low temperature is mainly caused by the inhibition of the development-related genes. The transcriptional response of seed and seedling to low temperature is different, and the differential expressions of genes in signaling and metabolism pathways may contribute to the chilling tolerance of HF during seed development. These results provide informative clues and will significantly improve the understanding of rice seed development regulation and the mechanism of cold response in rice seed.