The fire erupted not far from Highway 154 in an area where a fire in 1990 killed one person and burned about 550 homes.
Within a few hours, it had spread through about 25 acres of brush.
There were worries that afternoon winds would make it much bigger, but firefighters were able to stop its progress. It was 40 percent contained.
"The wind is not a factor yet. We're continuing to make water drops and trying to establish fire lines," County Fire Capt. David Sadecki told The Associated Press.
One firefighter suffered a minor leg injury, Sadecki said.
Television reports showed a dense plume of grayish-white smoke above the steep hills and a line of flame snaking through a canyon.
Air tankers painted the flames with orange-red fire retardant, and firefighters armed with hoses made stands near homes.
Four helicopters and two air tankers were aiding firefighters on the ground, who were supported by 16 fire engines and several hand crews.
At the fire's height, deputies went door to door telling people to leave the Painted Cave area, an isolated mix of cabins and homes near a rocky ridge. By mid-afternoon, about 40 homes had been emptied, but they were allowed back before nightfall.
The cause of the fire was unknown, but some power lines were down in the area, fire officials said.
Ganga White, founder and director of the nonprofit White Lotus Foundation yoga retreat, said he saw the fire erupt across the street, apparently from a downed power line.
The flames were 50 feet high but heading away from the retreat, he said.
Thirty-five people were attending a training session for yoga teachers.
"They're all packed up ... and they're all by their cars," White said. "We can be out of here in five minutes."
The area, about 75 miles northwest of downtown Los Angeles, is known as a fire hazard, especially when "sundowner" winds begin to blow through the canyons toward the ocean in the afternoon.