Because accretion and merger shocks in clusters of galaxies may accelerate particles to high energies, clusters are candidate sites for the origin of ultra-high-energy (UHE) cosmic rays. A prediction was presented for gamma-ray emission from a cluster of galaxies at a detectable level with the current generation of imaging atmospheric Cherenkov telescopes. The gamma-ray emission was produced via inverse Compton upscattering of cosmic microwave background photons by electron-positron pairs generated by collisions of UHE cosmic rays in the cluster. We observed two clusters of galaxies, Abell 3667 and Abell 4038, searching for very high energy gamma-ray emission with the CANGAROO-III atmospheric Cherenkov telescope system in 2006. The analysis showed no significant excess around these clusters, yielding upper limits on the gamma-ray emission. From a comparison of the upper limit for the northwest radio relic region of Abell 3667 with a model prediction, we derive a lower limit for the magnetic field of the region of ~0.1 Delta *mG. This shows the potential of gamma-ray observations in studies of the cluster environment. We also discuss the flux upper limit from cluster center regions using a model of gamma-ray emission from neutral pions produced in hadronic collisions of cosmic-ray protons with the intracluster medium. The derived upper limit of the cosmic-ray energy density within this framework is an order of magnitude higher than that of our Galaxy.