Short answer: Unlikely—unless you live atop specific geological formations.
Crystals typically form deep underground under precise temperature, pressure, and chemical conditions over thousands to millions of years. Quartz veins, calcite deposits, and gem-bearing pegmatites are subsurface features. Surface exposure usually requires erosion or tectonic uplift—rare in typical residential areas.
Consult local geology guides, state geological surveys, or rockhound clubs for promising public collecting sites. Many areas have designated fee-dig mines, public lands for hobby collecting, or known mineral localities. Random backyard digging yields mostly dirt—and broken sprinkler lines.