No, lip size does not directly affect your ability to play a musical instrument.
While some instruments (brass, woodwinds) require a specific embouchure—the way the mouth, lips, tongue, and facial muscles interact with the mouthpiece—this depends on the shape, structure, and flexibility of the lips, teeth, and oral cavity, not their overall size.
Many world-class players have lips of every description. Proficiency comes from:
If you're concerned about physical suitability, consult a qualified teacher—they can assess your individual anatomy and guide you to the right instrument and approach.