Note that if you do want to speak against it, "I don't like it because X" is not very useful to the zoning board. The board has a specific set of criteria by which variances are supposed to be evaluated. You want to be able to say "I believe that this variance should not be granted because it does not address requirement Y in the town zoning ordinance, and the same goals could be achieved without a variance and without significant hardship by doing Z."