The findings, released this month, offer a rare glimpse into brokers’ transaction costs—the spreads between the price you pay when buying options and the price you get when selling them. While brokers disclose how much they charge in fees, they shed little light on the costs associated with such spreads.

At Robinhood, nearly 7% of the dollar value of options transactions is swallowed up by such costs, far more than at rival brokers, according to the study by a trio of finance professors at the University of California, Irvine, and Washington University in St. Louis. The authors used their own money to carry out nearly 7,000 options trades at six brokerage platforms from March to June.

Even after accounting for higher fees at other brokers, Robinhood customers still get a bad deal on most options trades, the authors concluded. Vanguard was the best-performing broker in the study, followed by Fidelity Investments.

Robinhood disputed the findings.

In conducting the study, the professors repeatedly bought and sold options on 18 popular stocks and exchange-traded funds. They executed the trades simultaneously at each broker and compared the prices they got. The other three brokers they used were E*Trade, Charles Schwab and TD Ameritrade.

Options are contracts that let investors buy or sell a stock at a specified price, by a particular expiration date. Part of their appeal is that they can be used to place leveraged bets on stock prices. Individual investors have piled into options since the Covid-19 pandemic.

But unlike in stocks—where many brokers voluntarily disclose data on how well they execute trades, and regulators require the firms that execute investors’ orders to disclose similar statistics—there is little public data on transaction costs in options.

Brad Engime, a state employee and punk-rock guitarist in southern New Jersey, uses several brokers to trade options in his spare time. Robinhood is his preferred broker for simpler, more “gambly” bets, while he does more complex trades on Fidelity and Schwab’s Thinkorswim platform, he said.

After reading the study, Engime said he plans to shift more of his business from Robinhood to Fidelity. Brokers should disclose more about the hidden cost of options trades, he added.

“It would benefit people to have more transparency,” he said.

Options are a lucrative business for Robinhood. During the first half of this year, the company’s revenue from options was $336 million, more than one-quarter of its total, its financial filings show.

Robinhood profits from its customers’ option orders by collecting rebates from electronic trading firms that process the orders, a practice called payment for order flow. The study found that the more brokers collected on such payments, the steeper their hidden transaction costs. Vanguard was the only broker in the study that didn’t get paid for options orders.

Options prices are publicly quoted on exchanges, and the most visible measure of transaction costs is the bid-ask spread. That is the difference between the lowest price quoted by sellers on exchanges (the ask) and the highest price quoted by buyers (the bid).

Retail brokerages often try to fill customers’ orders at prices within the bid-ask spread. Getting orders filled in the middle of the spread benefits investors because it means they save money on purchases, or earn more money on sales. The study found that Robinhood was more likely than other brokers to fill orders near the extreme ends of the bid-ask spread.

Matt Billings, vice president of brokerage at Robinhood, said the study underestimated the degree to which Robinhood delivers better prices to its options customers, relative to prices on exchanges. “We are in touch with the authors about our concerns,” he said.

One metric that the authors examined, called the round-trip transaction cost, measured how much money they lost by buying an options contract and then promptly selling it. Ideally, this should be zero—what you get for selling should equal what you paid. In practice, it isn’t.

At Robinhood, the authors found their average round-trip cost was 6.8%, meaning that about $6.80 of every $100 they spent was eaten up. By comparison, the same metric was 1.8% at Fidelity. At Vanguard, it was a slightly negative number, -0.3%, a sign that Vanguard was filling orders at prices even better than the middle of the bid-ask spread.

So does that mean all options traders should switch to Vanguard? Not necessarily. Vanguard charges a $1 fee for each options contract you trade. E*Trade, Fidelity, TD Ameritrade and Schwab charge 65 cents a contract. Robinhood doesn’t have such charges, although it collects various regulatory fees that add up to about 3 cents a contract.

When factoring in fees, Robinhood is the cheapest broker for options with a bid-ask spread of 1 cent, the authors found. But for options with wider spreads—the majority of options contracts in the marketplace—Fidelity or Vanguard are the best, they wrote.

Write to Alexander Osipovich at alexo@wsj.com