Small businesses are losing ground in the hiring game.

Head counts at companies with fewer than 50 employees declined in three of the past four months, according to ADP payroll data, even as employment at larger firms continued to grow.

Owners of many small companies say inflation has added to the pressures of an already tight job market, making it increasingly difficult to keep pace with the wages and benefits offered by large employers. The hiring challenges are stunting growth, small-business owners say, and further clouding their deteriorating economic outlook.

At Gahlsdorf Logging Inc., the head count is stuck at around 20, making it difficult for the Rickreall, Ore., logging contractor to keep its eight-person crews fully staffed. Owner Jim Gahlsdorf said he turns down an offer about once a month to look at or bid on a potential job because he doesn’t have the workers.

Some small logging operators have dropped to one crew from two in response to labor shortages. Mr. Gahlsdorf said he worries that a slimmed-down operation wouldn’t produce enough revenue to cover overhead. He is hoping that as the number of crews operated by competitors declines, customers will pay more for his services, allowing him to pay his workers more.

Gahlsdorf Logging has had to turn down chances to bid on jobs because the Oregon-based company can’t find enough workers, its owner says.

Photo: Celeste Noche for The Wall Street Journal

“If we keep retreating from this problem, we will never solve it, and that will be the end of my business,” Mr. Gahlsdorf said. “If we just hang on, we will be in a better spot.”

Sixty-three percent of small-business owners say that hiring challenges are affecting their ability to operate at full capacity, according to a June survey of more than 825 small businesses for The Wall Street Journal by Vistage Worldwide Inc., a business coaching and peer advisory firm.

Some large companies that saw significant growth during the Covid-19 pandemic are beginning to take a more cautious approach to hiring, which could over time create more opportunities for smaller employers to bolster their ranks. For now, however, “small firms are still playing catch up,” ADP chief economist Nela Richardson said. If the economy weakens, small firms are also likely to change their hiring plans, she added.

Small-business confidence continues to fall, dropping in June to lows last seen in July 2020, the Vistage data show. Nine percent of small-business owners expect U.S. economic conditions to improve over the next 12 months, down from 12% in May and 53% in June 2021. Despite their economic worries, 52% of small-business owners expect head counts to increase in the coming year, the survey found.

Like companies of all sizes, small businesses are spending more to attract and retain workers. Seventy-six percent of small-business owners said they had boosted wages in response to labor-market challenges, according to the Vistage survey, while 44% reported adding employee benefits.

Jim Gahlsdorf, owner of Gahlsdorf Logging, at a work site in Lincoln County, Ore.

Photo: Celeste Noche for The Wall Street Journal

Mr. Gahlsdorf’s logging company has raised starting pay by about $5 in the past year. A recent Facebook ad offered starting pay of $25 to $26 an hour for “choker setters,” who fasten steel cables around logs. “[W]e will train the right person!” the ad said. “No drugs. Show up. Learn. Work Hard. Get Logs. Make $$.”

Black Earth Compost, an organic-waste collection and compost-processing company in Manchester, Mass., with 78 employees, starts drivers at $18 an hour. Last year, the company began offering a $3-per-hour bonus to drivers who start on time, make all their pickups and deliveries and meet other performance targets.

“That stopped the hemorrhaging, but it’s been tight ever since,” said owner Conor Miller, who recently brought in someone to help with hiring and has increased company benefits. The company added a 401(k) retirement savings plan last year and now allows employees to receive a portion of their pay in the form of bitcoin, which until the recent volatility in the cryptocurrency market was a big draw for younger workers.

Goulding & Wood Pipe Organ Builders, in Indianapolis, has looked for months to hire a replacement for an employee tasked with installing control systems and wiring consoles.

Photo: Monty Thurman/Goulding and Wood

Small-business owners often pitch in themselves when staffing shortages arise. At Goulding & Wood Pipe Organ Builders, which builds and maintains pipe organs, the employee responsible for installing control systems and wiring consoles quit in November after four years on the job. The Indianapolis-based company, which employs 13 people, still hasn’t found a replacement.

“I’ve pretty much taken over most of the responsibilities,” said owner

Mark Goulding, who has raised wages twice since the fourth quarter of last year. “It’s why I am in Grand Rapids, Mich., today, finishing a job he should have done.”

The growth of remote work has made it easier for large employers to poach workers in faraway locations where the pay scales may be lower. Health Designs, a workplace wellness company, based in Ponte Vedra Beach, Fla., recently lost an executive to a company based in Boston and another employee to a Canadian company, both offering remote work and higher pay.

“We are no longer competing with other local companies in local geographies. We are competing against the world,” said Health Designs Chief Executive Chris Margolin, who recently filled one position but still has two openings in his 18-person home office.

Freeway Towing, in Monterey Park, Calif., has struggled to hire more workers even though it has raised pay and productivity bonuses, the company’s owner says.

Photo: Yousif Haddad

Staffing gaps can take a toll on customer service. Freeway Towing Inc., in Monterey Park, Calif., has 46 employees, down from nearly 70 before the pandemic. The company, which provides towing, repairs and accident clean up, would employ 100 or more people if finding workers wasn’t such a challenge, said co-owner

Nadia Haddad.

It now takes up to three hours for one of the company’s 67 trucks to arrive at an accident scene or pick up a stranded motorist, up from a more typical 20 or 30 minutes when the company is fully staffed, Ms. Haddad said. Angry motorists also call more to complain, which has led to more turnover among the office staff.

“The average office staff person in my company would stay for 10 to 12 years,” said Ms. Haddad. “Since the pandemic, I’m lucky if I keep them for two years.”

The company has boosted hourly wages and productivity bonuses and now pays for employees to be certified as Class A drivers, allowing them to operate wreckers, cranes and other heavy-duty vehicles. But it can’t match larger companies when it comes to paid time off and other benefits, she said.

“The pay is good. The hours are flexible. What we do for a living is rewarding,” said Ms. Haddad. “It’s just that the competition is magnificent. It’s significantly more magnificent than me.”

Gahlsdorf Logging might not generate enough revenue to cover overhead if it were to operate with fewer crews, its owner says, citing labor shortages.

Photo: Celeste Noche for The Wall Street Journal

Write to Ruth Simon at ruth.simon@wsj.com