If a predator bites a bird or mammal then the animal is likely to die. A poison is unlikely to save the organisms life. Only a very few birds can be poisonous depending on what they eat and while there are a few venomous mammals none are poisonous. (Venom is actively injected into another animal, for example a snake's bite, or a bee sting. Poison is dangerous if you eat or touch the organism for example a poison dart frog.)
A poisonous bird or mammal might through its poison save the lives of its relatives, perhaps even its offspring. So the poison would confer some evolutionary advantage on the bird or mammal, but developing an immunity is likely to provide even more of an advantage for the predator. So in an evolutionary arms race, the predator is more likely to win with a bird or a mammal than with a plant or colonial animal.
The predators apparently evolved immunities faster than the birds or mammals evolved poisons. On the other hand for the colonial animal or plant, the prey survives because of the poison. So the prey might win the evolutionary arms race. All of this helps to explain why plants and colonial animals are so commonly poisonous and birds and mammals so rarely poisonous.
There are many examples of this phenomena. Plants are like colonial animals in that removing leaves or a branch does not necessarily kill the organism. So this hypothesis suggests they should be poisonous, and indeed all or almost all plants are. Humans are immune to many of their poisons, herbivores, like deer, are immune to many more plants, but strict carnivores, like cats, will die if they eat many of the plants deer and even humans safely eat.
Colonial animals, for example, sea anemones, jellyfish, coral, and most sponges are poisonous. Bees, wasps, and ants are also colonial and they also have a poisonous sting. The colonial insects are very active while sponges, coral, and sea anemones are not. So they are very different, but the fact that both are colonial means they both use poison or venom.
Starfish and other members of the phylum echinoderm are often poisonous. Like the colonial animals, if they suffer injury, they can regrow damaged parts. Unlike many mammals and birds they do not rely on flight or speed to survive. So the first bite of the predator does not doom the animal. If the predator quickly realizes its mistake the animal can survive.
It is often asked, why aren't all animals poisonous. This hypothesis may provide part of the answer. It is not the whole answer. Obvoiously, there are many animals that are poisonous and are not colonial.