The historical term is consort—a group of instruments playing together, especially in Renaissance and Baroque music. You'll encounter:
As ensembles grew and standardized in the Classical era, larger mixed groups became orchestras. Smaller standard combinations acquired specific names:
"Consort" today usually signals early music specialization (e.g., Baltimore Consort, Viol Consort, Newberry Consort). For post-1750 ensembles, the specific instrumentation name (quartet, quintet, octet, chamber orchestra) is standard.
So: Renaissance/Baroque group = consort. Classical+ group = [instrumentation] + [size name].