If you are sitting second wheel (that is, there is only one person in font of you), and the person in front of you (who is leading the pack) pulls off to the side, you have two options. You may 1) continue pedaling, slightly increasing your effort to maintain the same speed, and then continue to pull the pack along for a bit or 2) continue pedaling, slightly increasing your effort to maintain the same speed, and then immediately pull off in front of the person who had previously pulled off in front of you.
Note that neither of these options includes slowing down in order to stay behind the person who just attempted to pull off. This kind of action (doing everything possible NOT to go to the front) may be acceptable in last few kilometers before the final sprint, but otherwise, there is no excuse for it when sitting second wheel. If, for some reason, you really dont want to pull through, you should simply not be riding second wheel. I dont know how many times this occured last night but it was pretty frustrating, and forced the same couple of guys up to have to go around multiple people to move back up to the front to keep the pace respectable.
Group rides can be great for training, especially when contain a large number of fast riders who force you outside your comfort zone. But if that is the case, then by all means, let them drive the pace and stay out of the way. And if you are bored in the back? Great! But just be ready to move all the way to the front and do your share of the work.