Will the NBA make Los Angeles Clippers owner Donald Sterling the first to give up his team?

(CNN) -- Adam Silver has been NBA commissioner for only a few months. A few relatively controversy-free months.

Now -- with the longtime owner of a top team alleged to have made racist remarks in a phone conversation posted on a celebrity-focused website -- Silver is stepping down a new road.

Only one owner has ever been suspended, in 68 years. That was Minnesota's Glen Taylor in 2000, when the Timberwolves made a secret deal with a star player to circumvent salary cap rules.

The league has never forced an owner to give up his team.

Should Los Angeles Clippers owner Donald Sterling face either of those punishments?

First, the NBA says, it will investigate the matter -- see if the recorded conversation is authentic, if it is actually Sterling, if it has been edited or manipulated.

"All members of the NBA family should be afforded due process and a fair opportunity to present their side of any controversy, which is why I'm not yet prepared to discuss any potential sanctions against Donald Sterling," Silver told reporters Saturday.

Silver said the league will move "extraordinarily quickly" in its investigation and he hoped to have an initial report in "the next few days."

He indicated the NBA had been in contact with Sterling already, saying the league's longest-tenured owner had agreed not to travel to Oakland, California, where the Clippers will play the Golden State Warriors on Sunday in the first round of the NBA playoffs

There is no clear answer as to what the discipline should be, if the accusations against Sterling -- who has yet to comment publicly -- are true.

According to the website TMZ, Sterling made the alleged discriminatory remarks during a 10-minute argument he had with girlfriend V. Stiviano on April 9. TMZ posted a recording of the argument online Friday.

If authentic, the remarks seem to reflect Sterling's embarrassment and frustration with Stiviano over her associating with African-Americans at Clippers games and for posting such pictures on her Instagram account.

"This is a failure of leadership on so many levels," Jason Johnson, an HLN contributor and a political science professor at Hiram College, said. "This is not the first time we have had an owner get in trouble, and they should have had rules in place to deal with these kinds of situations."

The most likely penalties, said CNN Sports anchor Andy Scholes, are a fine and a suspension, possibly banning the 80-year-old Sterling from team operations for as long as a year.

TNT analyst Charles Barkley said no question, Silver should suspend Sterling soon.

"He's got to suspend him right now," Barkley said Saturday night. "First of all, they have to prove it is his voice on the tape, but this is the first big test of Adam Silver. You can't have this guy making statements like that. He has to suspend him and fine him immediately."

Sterling has never been sanctioned by the league since he bought the team in 1981.

Other owners have been fined, most notably Dallas Mavericks owner Mark Cuban, who has paid out nearly $2 million in 19 fines, according to Bleacher Report. One fine reportedly was $500,000, about the same amount Miami Heat owner Micky Arison supposedly had to pay in 2011. But those were for remarks about officiating (Cuban) and the lockout (Arison).

And a similar record-level fine for the Clippers owner might not seem like much to the public, since Sterling is worth billions.

The league probably cannot force Sterling to give up the team, which according to Forbes magazine is worth $575 million, Scholes said.

According to Scholes, the NBA constitution is confidential, and reportedly the league would only step in to remove an owner if the team were in financial trouble.

The value of the Clippers, who were a terrible team for decades, has been on the rise with their recent success on the court. Led by all-stars Blake Griffin and Chris Paul, who is also the head of the players union, the team has set wins records in the past two seasons. The other players will look to Paul for guidance, but a players walkout is unlikely.

"They are looking at Chris Paul and they are trying to see what can they actually do to show their disdain and dislike (for Sterling)," Johnson said.

The NBA will face an image problem if Sterling turns out to be the voice on the recording, in which a man tells a woman that she shouldn't be seen at Clippers games with Earvin "Magic" Johnson because he is African-American. The woman has been identified as Sterling's girlfriend V. Stiviano.

The question would become: Could the Clippers sign any free agents in the offseason if Sterling remains the owner?

LeBron James, who could potentially be one of those free agents, indicated he wouldn't consider playing for Sterling's Clippers.

"There's no room for Donald Sterling in the NBA," he said Saturday before the Miami Heat played the Charlotte Bobcats. "There is no room for him."

James told ESPN he was certain Silver would take care of the matter.

Jason Johnson said Silver needs

to be forceful.

"He can say, 'Look, you have damaged the brand. You have damaged the league, and consequently, I think we are going to have to look at your right to own this team,'" Johnson said.

The NBA playoffs will continue through mid-June, when the league will crown a champion.

™ & © 2014 Cable News Network, Inc., a Time Warner Company. All rights reserved.