Their move, first reported in the Financial Times, marks a huge blow to the social network’s plans to launch what it envisions as a global currency.
Latin American payments company Mercado Pago also announced it was leaving.
Visa confirmed they had pulled out, adding: “We will continue to evaluate and our ultimate decision will be determined by a number of factors, including the association’s ability to fully satisfy all requisite regulatory expectations.”
Facebook’s head of the project David Marcus said people should not be “reading the fate of Libra into this update”, although he admitted it was “not great news in the short-term”.
The Libra Association now has no major payments companies as members, meaning it cannot count on a global player to help people turn their currency into Libra and facilitate transactions.
The remaining members are mainly venture capital, telecommunications, blockchain and tech companies.
But Libra will press on, according to Dante Disparte, its head of policy and communication.
He said: “We are focused on moving forward and continuing to build a strong association of some of the world’s leading enterprises, social impact organisations and other stakeholders.
“Although the makeup of the Association members may grow and change over time, the design principle of Libra’s governance and technology, along with the open nature of this project ensures the Libra payment network will remain resilient.”
Facebook announced in June that it was establishing the group, with the aim of launching the digital currency in June 2020 but it quickly ran into regulatory issues.
France and Germany have pledged to block Libra from operating in Europe and backed the development of a public cryptocurrency instead.
At its unveiling, Facebook said it wanted to allow people to make payments with the currency via its own apps, as well as on messaging service WhatsApp, while partner firms would also be able to accept Libra for transactions.
Paying, it said, would be as easy as texting.
But concerns have grown over how people’s money and data will be protected, as well as over the potential volatility of the currency.
Facebook chief executive Mark Zuckerberg is expected to discuss the project when he testifies before the US House financial services committee on 23 October.