You almost had it. Racism ends when white people no longer benefit from it or feel they need it. All of the civil rights marches and legislation mean nothing if white people still carry hate and malice in their hearts.
That is what you got wrong: who can end this suffering.
For example, homophobia ends when straight people no longer feel they need it. Misogyny is finished when men feel they no longer need to subjugate women to feel powerful or “manly.”
The oppressed does not determine when the suffering ends, the oppressor does. That is the flaw with your “baloney.”