The really toxic zoas are only certain types of palys from what I understand - not all zoas and palys are as highly toxic as those certain types. But, it's good to wear gloves with all of them, and to be sure to wear safety glasses & keep your mouth closed.
Most, if not all corals have some type of defense - stinging and/or toxins. Avoid using cut hands with no gloves on, and watch out - some like the zoas and mushrooms can squirt you in the eyes or mouth.