The flotilla, consisting of about 38 fishing vessels and some other boats, was "corralled" by Royal Gibraltar Police and military boats after leaving Spanish waters near an artificial reef created by the Gibraltan government.
The reef is at the centre of a row between Britain and Spain, which escalated after Madrid introduced additional checks at the border for those entering or leaving Gibraltar and seen workers and tourists face queues of up to five hours to pass border controls.
Chief Inspector Cathal Yeats, of the Royal Gibraltar Police, said the boats first met in Spanish waters and, despite efforts by British boats to stop them, crossed into Gibraltan waters before being ushered out again.
Yeats said: "At around 9am about 38 Spanish fishing boats and seven or eight pleasure craft converged in the area of the western anchorage.
"We had our own police cordon along with Royal Navy and other assets and we corralled them in the area of the south mole.
"They tried to breach the cordon several times but they were not successful."
The boats left Gibraltan waters at around 11am.
Police were not surprised by the protest, which had been planned since Friday and reported that it had passed peacefully, with no arrests.
The Spaniards, who set out from the Campo de Gibraltar in the country's south, maintain the reef impedes upon their right to fish, while Gibraltar says the reef's creation was necessary to protect local fish stocks.
On Friday Prime Minister David Cameron raised the imposition of the extra checks with European Commission president Jose Manuel Barroso.
In a telephone call, he underlined Britain's belief that the checks were "politically motivated and disproportionate" and therefore contrary to the EU right of free movement.