There are two things to keep in mind before I get into the idea of animal spellcasters. The first is "how does spellcasting function?" If it is just memorization of actions, sounds and materials, then animals should be able to learn basic magic. The other is a bit of wisdom from a friend of mine- "rules are fo players", meaning GMs should change or ignore the rules if it improves play. In this case allowing animal spellcasters means dropping the intelligence score minimum to 1 or 2. Or one could always rule that the god of animals (and plants and vermin, see below) simply changed the rules so that their creations have an easier time surviving a world dominated by sapients such as humans and dragons.
Animals that use magic might be very common in a setting as long as magic is just actions, sounds and materials. If an ape learns a spell by inventing it by accident, it can teach its offspring the spell. Over time different bands of apes learn different spells and refine them over the generations. In this case it is behavioral evolution. Wolves that "dance" to improve the hunt, fish that form glyphs with their schools and whales that sing bard (ish) songs are all possible.
Plants and vermin can rely on a very different kind of magic, one that is also drive by evolution. They can have symbols, sigils, glyphs, runes and other magical writing either embedded in their form or, for those that can move, carved into the environment. Ants, termites and bees could have very complex hives where different parts provide different magical effects. They combine into something potentially much greater than their parts (i.e. metamagic). Trees could grow symbols in their bark or leaves. Those plants and vermin that use the magic and thrive are going to outsurvive and outbreed those without such powers.
Using these ideas might change the rules for other creatures. Take hook horrors for example. They are monstrous humanoids with an intelligence of 7. They could easily learn how to carve stone with symbols of power and pass such information on to their offspring. They could use magic for hunting or defense but they might also learn utility magic that allows them to survive in larger and larger communities. I have never been much of a fan of hook horrors until I watched The Dark Crystal a few years ago. Seeing the garthim gave me a few ideas on how to use horrors more effectively as well as a possible place in a setting. The symbology used in The World of the Dark Crystal (an excellent book if you can find it) merged with the idea of horrors using magic and gave me rune using horrors. What I did in those notes is give them the ability to activate the runes from a short distance (5-25') depending on how powerful the spell was. I also came up with an idea for crabmen, that in their 250-325 million years of existence, they could learn and refine magic even with their rather low intelligence. Crab magic is so advanced and different from that of humanoids that humans simply can not learn from it (again metamagic came into play).
But what kinds of spells can animals learn, appear as glyphs on beetle exoskeletons and runes on bark? I would allow any cantrip through second level that improves survival. The younger the species* or older the teachings, the more powerful the effect. Mage Armor may be the most common followed by Resistance, Endure Elements, Detect Poison, Ghost Sound, Create Water, Daze, Cure Minor Wounds and Open/Close (the last being used to get into nuts, eggs and shellfish). Between Create Water and Endure Elements, tropical plants could colonize the deserts of the arctic regions. Daze might be very common for predators and prey. Cure Minor Wounds increases the lifespan of many creatures. And as these species become more and more reliant on their magic, their forms, their phenotypes to be technical, will change. Beetles that use magic to stun prey may have reduced mandibles and shorter legs. Plants that use Create Water may have shallow roots that spread out and are used for balance rather than going deep into the soil. Wolves might use a weakened form of Fabricate that only functions on dead wood to create dens (or forts?) that keep the weather out.
The irony of this, at least for Animals, is that it doesn't change their type. They are not Magical Beasts, just spellcasting Animals.
* The reasoning for this is that newer species will have more refined magic than older ones. Which isn't always true considering the evolution of parasites, so take it with a grain of salt.
