We were never asked to be vegetarian; we were only asked to observe a few main principles. Refraining from eating meat was just one of the interpretations.
I'm sure many, if not all, of the highest Buddhist clerics, as close to enlightenment as we will ever know on Earth, are vegetarian. But the most hypocritical, delusional, dark-hearted person I know, also happens to be vegetarian and aligns himself with "Buddhism".
You're not any more right, noble or compassionate than anyone else because you are vegetarian or identify yourself under a banner or label. You will be more right, more noble, more compassionate than others when you are more right, more noble, more compassionate than others.
We'll harm many things in our lifetime. Eating requires killing, whether it's an animal or a plant. Plants are living organisms too. They satisfy all the criteria: they move on their own, they grow, they respond to stimuli, they reproduce, they consume nutrients, they expel waste, they have cells that demonstrate specific cell grouping/organization.
Some cultures don't consider eating certain animals to be eating meat. In many places, fish are not considered "animals" as land animals are. They're not vegetables either, but traditionally in Vedic cultures and many Asian cultures via the spread of Buddhism, fish don't have higher intellect, "a third eye" or the spiritual centre that other animals have. This is why many vegetarians will eat fish and insect meat. Their culture and belief system are not wrong for classifying living things differently and they certainly don't deserve the narrow-minded snarking I've seen.
Every step you take, there's a chance you'll trample an insect or some living creature. If you live in a house, apartment or any kind of modern structure, you've contributed to the demand that involved trees being clear cut and forest animals being displaced. No matter how careful you think you are, how participatory or non-participatory you are in whatever way, in whatever society you choose to live in, you'll always be a part of a long chain of events that involves killing and destroying at some point. Even if you live completely "off the grid", you need to eat, you need to survive. That's always been the natural cycle of life.
Use what you need, but be modest about your needs. Be mindful of the origins of your resources and consequences of your usage. And if those criteria are satisfied to the best of your abilities, then there's no sense worrying about things that are truly not yours to worry about.