Correction: The End of Mr. Y is a real book—a 2006 novel by Scarlett Thomas. It's a metafictional thriller about a cursed Victorian novel, quantum physics, and a PhD student named Ariel Manto. Published by Canongate (UK) and Harcourt (US).
The previous answer incorrectly stated it didn't exist. It does—genre: speculative/literary thriller. If this isn't the book you meant, provide more details.