Preheat the oven to 375˚F. Line a few baking trays with parchment paper and set them aside for later.
Add the flour, cornmeal, sugar, baking powder, baking soda, and salt to a large bowl. Mix the ingredients together with a whisk until combined.
In a separate bowl, mix together the egg, buttermilk, olive oil, chopped rosemary, lemon zest, and orange zest. Pour the wet mixture into the bowl of dry ingredients.
Mix everything together with a rubber spatula until well combined. For each whoopie pie, scoop about 1 1/2 tablespoons of batter onto the prepared baking sheets and spread the batter out into a circle 1 1/2 inches in diameter. Space each whoopie pie 2 inches apart.
Bake the whoopie pies in the oven for 10-11 minutes. When a toothpick is inserted into the center of a whoopie pie it should come out clean. Transfer the whoopie pies to a cooling rack to cool completely.