Jacksons Landing is set to be demolished in weeks despite widespread support from people to transform the building.
More than 800 people signed a petition by independent councillor Paul Thompson to keep the building which Hartlepool Borough Council paid £1.5 million for in 2013.
But councillors voted in favour of knocking down the former shopping mall at a cost of £40,000.
There is no fixed date for the demolition but it is expected to start within weeks.
Thousands of music fans will fill the building today when it hosts the second annual We Are Family music festival.
Independent councillor Paul Thompson tried to retain the building after organising an online petition supported by more than 800 people.
Hartlepool businessman Chris Middleton also launched a Facebook page called Save Jackson’s Landing.
The demolition was supported by Labour and Conservative councillors and opposed by UKIP, Putting Hartlepool First and the Independents.
Council leader Councillor Christopher Akers-Belcher said demolition would make way for a future development – still to be decided – that will make Hartlepool a tourist destination and complement the National Museum of the Royal Navy (Hartlepool).
He said: “It very much articulates our ambition for the site and the long-term prosperity of Hartlepool.
“Jacksons Landing as a building has become associated with failure and Hartlepool is worth much more than that.
“The building dominates the site and restricts the potential for new development.”
The council bought the site for £1.5 million using an interest-free loan in 2013.
It has been empty for more than 10 years after it ceased trading in 2004.
Coun Akers-Belcher said the site had become a “hot spot” for anti-social behaviour over the summer with the police recording criminal damage, break-ins and youths climbing on the roof.
Coun Thompson accused Coun Akers-Belcher of being “disingenuous” over the figures saying the problems were not limited to Jacksons Landing but to the wider dock.
Coun Thompson cited hundreds of comments made on social media from people who were against demolition and suggested they were fed into a consultation process being led by a team of architects. The proposal was lost on a vote.
Coun Thompson said: “I have yet to be presented with any factual evidence that knocking it down is in the best interests of the development or that all options have been explored.”
“Ok there’s a vision, but there’s no concrete plan. The truth is we don’t have a plan.
“The pace this is happening gives me further cause for concern.”
He added he understood talk of a hotel operator interested in the site was in the very early stages.
But Coun Dave Hunter claimed such a deal could create around 70 jobs.
Coun Hunter said: “That piece of land is vital to the future of this town. Getting the National Museum of the Royal Navy was a phenomenal achievement and we have got to support it.”
Coun Shane Moore of UKIP said: “If a developer genuinely wants to come along and flatten it and build something fit for purpose let them, but let them do so at their own expense, not ours.”
He urged any new scheme does not include plans for any new homes saying there is already a large number of empty properties in the marina area.
Conservative group leader and property developer Coun Ray Martin-Wells said: “The site has been widely advertised. We have tried to market it for three years and we have got no successful buyer.
“It is for that reason to secure the future of the site we Conservatives support this.”
Coun Akers-Belcher said the £40,000 will come from the uncommitted funds left over from last year’s budget.