Ukraine said on Thursday that Russia had gathered 106,000 troops and 1,500 tanks near its border and demanded a “verified withdrawal” of Russian forces, while Moscow warned that it would “take all necessary measures” to ensure its security, as a third round of military security talks in Eastern Europe failed to produce a breakthrough.

Russian officials did not close the door on diplomacy, but instead struck a more pessimistic tone. In a television interview, Russia’s deputy foreign minister, Sergei A. Ryabkov, said it was premature to convene more talks until the West abandoned what he called a “dead-end” approach. The US reiterated in Vienna that it was ready “for an open dialogue,” but that it would “reject blackmail and never allow aggression and threats to be rewarded.”

The exchanges at an international meeting raised tensions and implicitly threatened further Russian military intervention in Ukraine. The threat posed by Western military activity near its borders, particularly in Ukraine, has reached an unacceptable level, according to Russia, who has demanded immediate “security guarantees,” including NATO’s agreement to reduce its presence in the region.

Ukraine, taking part in this week’s diplomacy for the first time, said Russia’s troop buildup needed to be reversed. “The Russian leadership demonstrates once again Moscow’s willingness to point the gun at our common European security at any time,” Ukraine’s representative, Yevhenii Tsymbaliuk, said.

Thursday’s talks were held at a lower diplomatic level than those held earlier this week in Brussels and Geneva, with no one above the ambassador rank in attendance from the key countries involved. They were held at a regular meeting of the Organization for Security and Cooperation in Europe, a 57-country body that includes Russia, Ukraine, and the United States, and is expected to serve as a key venue for additional negotiations if the Kremlin decides to pursue its goals diplomatically.

“It appears that the risk of war in the OSCE area is now greater than it has ever been in the last 30 years,” Poland’s Foreign Minister Zbigniew Rau said in opening the session. This year, Poland assumed the organization’s rotating chairmanship.

Despite representing Poland, one of Russia’s most vocal European critics, Mr. Rau appeared to extend an olive branch to the Kremlin while implicitly criticizing President Vladimir V. Putin for attempting to re-establish a sphere of influence in the region.

“We are not apathetic to security concerns raised by participating states,” Mr. Rau said. “I believe the O.S.C.E. is the appropriate forum for discussing all aspects of comprehensive security.”

It was the latest sign that Western countries are scrambling to engage with Russia, which has threatened a “military-technical” response if security concerns, such as Western military cooperation with Ukraine, are not addressed.

While Russia denies any intention of invading Ukraine, researchers have discovered new evidence of Russian troops moving toward the Ukrainian border in recent days.

However, it was unclear whether Russia would be willing to engage in diplomacy. Russian officials have stated that they will decide on next steps following this week’s talks. Sergey V. Lavrov, Russia’s foreign minister, said Moscow was awaiting a written response from the United States.

“The hard reality right now is that we were promised a written reaction,” Mr. Lavrov said in an interview aired on Thursday, according to the Interfax news agency. “We will wait for it. And then we will determine our next steps.”

Russia’s representative at the talks on Thursday in Vienna, Aleksandr Lukashevich, underscored that while Moscow was not ruling out the possibility of further negotiations, it was prepared to make such a determination in the coming weeks. Military analysts have noted that were Russia to invade Ukraine, the wintertime frozen ground would be advantageous to its heavy armor.

“If we do not hear a constructive response on our proposals in a reasonable time frame,” Mr. Lukashevich said in remarks released by his office, “we will be forced to draw the corresponding conclusions and take all necessary measures to assure the strategic balance and removing unacceptable threats to our national security.”