The rules governing Wall Street generally force stockbrokers to seek out the best prices for clients who pay them to buy and sell shares.


In recent years, though, brokers have had another enticement that can pull them in a different direction: payments from stock exchanges in return for sending them business.


The practice has attracted criticism from several industry participants and former regulators who say the so-called rebates that the exchanges pay Wall Street firms could give those firms an incentive to profit at the expense of investors. Now a new study using industry data says that the rebates could be costing mutual funds, pension funds and ordinary investors as much as $5 billion a year.


The 75-page study, being released this week, was written by Woodbine Associates, a financial consulting firm that does business with players on all sides of the issue. Woodbine said the report was done independently, without support from industry participants.


Some financial firms criticized Woodbine’s calculations and said the cost to investors was overblown, but did not dispute that the potential for a conflict of interest exists.


The study estimates that investors lost an average of four-tenths of a cent on each of the 1.37 trillion shares traded last year because of orders being sent to exchanges that were not offering the best final price. Stocks are sent to exchanges with inferior prices for reasons other than rebates, and the study’s tally includes those losses, but the authors say that the primary reasons for bad routing decisions are the rebates.


In one hypothetical situation, a mutual fund might ask its broker to buy one million shares of a major company. The broker sees that one exchange has a seller willing to part with the shares for $100 while another exchange has a seller offering $100.01 but is also offering the broker a tenth of a cent rebate per share. The mutual fund could end up paying $10,000 more than it needed to, while the broker would keep the $1,000 rebate.


The report puts a new spotlight on one of the most controversial practices that has sprung up as a growing number of exchanges have battled for the business of high-frequency traders and banks.


The losses to investors are usually gains for those high-speed trading firms and banks that factor the rebates into their automated trading strategies, and who seek out the trades of less speedy and informed traders.


In March, another financial consulting firm, Pragma, issued a report that drew attention to the conflict of interest created by rebates. Last fall, Jeffrey Sprecher, the head of the IntercontinentalExchange for futures and options, said at a conference that he would “have the regulators outlaw maker-taker pricing,” another name for the rebate system.


The most prominent criticism came in a 2010 report by two former chief economists at the Securities and Exchange Commission who said that “in other contexts, these payments would be recognized as illegal kickbacks.”


One of those economists, Chester Spatt, said that his group’s report initially generated conversation among regulators but was then overshadowed by the so-called flash crash of May 2010, in which stocks experienced a sudden and irrational plunge. The S.E.C. raised questions about the rebates offered by exchanges in a policy paper in 2010 but has not acted on it since.


Mr. Spatt said in an interview that the problem caused by rebates has not gone away and has most likely intensified as other sources of revenue for brokers have shrunk.


“Presumably many are acting in a self-interested fashion, and the self-interest leads to a lot of distortion,” said Mr. Spatt, who is now a professor at Carnegie Mellon. The report was sponsored by the trading firm Knight Capital.


Even before being officially published, the figures from Woodbine sparked a debate about the proper way to calculate whether clients were getting the best price in a given trade. According to Bill Conlin, the chief executive of the independent brokerage firm Abel Noser, the Woodbine report includes only one of the many costs that determine whether a broker’s client makes money, and that cost may not be the one that hurts investors most.


“There are little pennies here and there all along the line. They do add up,” Mr. Conlin said.