Ralph Fiennes hasn’t had to prove his range as an actor for some time, but now he seems determined to demonstrate his flexibility as a director, too. Fiennes made his directorial debut in 2011 with a bloody adaptation of Shakespeare’s Coriolanus that audaciously employed imagery from still-fresh conflicts in the former Yugoslavia. Fiennes anchored his bold film with an equally bold performance, playing Coriolanus as a man at home only on the battlefield, and ill-equipped for public life. The Invisible Woman is unlike Fiennes’ debut in almost every way. Set largely in hushed, shadowy interiors where lamps and candles fight a losing battle with darkness, it’s a film of whispers and repressed desires, and of what happens when people act on those desires. Fiennes plays Charles Dickens as a man seemingly never more comfortable than when he’s in public and playing the role of a literary celebrity. It’s the other roles that give him trouble.

Specifically, the role of husband. Fiennes’ Dickens is a playful father, but during the period covered by the film, his marriage has become a disaster. His wife Catherine (Joanna Scanlan), has borne him 10 children. In the process, she’s retreated to the margins, sitting and watching as her husband soaks in the adulation at public events. Dickens, along with his friend and collaborator Wilkie Collins (Tom Hollander), are staging a play for charity, one in which he’s cast pretty 18-year-old actor Ellen “Nelly” Ternan (Felicity Jones), one of three daughters from an acting family presided over by Frances Ternan (Kristin Scott Thomas). She isn’t brilliant in the part, but Dickens is happy to make her acquaintance—and compelled to continue it once the production has ended.

Working from Claire Tomalin’s 1990 book The Invisible Woman: The Story Of Nelly Ternan And Charles Dickens, which sought to shed light on their relationship, Fiennes portrays how even within the bounds of Victorian society, an intimate relationship could develop through lingering glances and stray moments. It also portrays how it had to be allowed to happen. Amid financial difficulty, Frances recognizes the advantages of pairing her daughter off with Dickens, even though he’s already married. Some of the strongest moments in The Invisible Woman show how passion slips through the cracks of the era’s social codes, whether it’s the growing attraction between Dickens and Nelly, or a talk between Nelly and Catherine that begins in niceties and ends with both women understanding where they stand without spared feelings or raised voices. It’s similarly good at portraying what isn’t allowed. Though already well along the way to becoming Dickens’ lover, Nelly is shocked that he takes her to the house Collins shares with a woman who isn’t his wife. Then there’s the way loose talk leads to rumors of the sort that could do real damage to a famous author’s reputation, a reputation on which Dickens depended for his livelihood.

Fiennes’ performance looms large over The Invisible Woman, but his film is ultimately less about Dickens than about living in Dickens’ shadow. The action plays out across a couple of time periods, opening and closing with Dickens several years dead and Nelly a married woman in her late 30s. Those who know her are aware she has some connection to the famous author. They talk freely of their admiration for his work, and Nelly has gotten good at replying to their compliments without revealing how well she knew the man, and how much more complicated he was than his genial image suggests. Yet through it all, she seems a little absent. And though the title refers to the way Nelly was kept out of public view during her time with Dickens, it might just as well refer to the way their relationship consumed her formative years and continues to cast a shadow well beyond them. Jones delivers a quietly wrenching performance as a woman who comes to recognize too late how much of herself she’s lost.

It’s subtle work in a film that is sometimes content to be a little too subtle. Fiennes—who just appeared as Magwitch in Mike Newell’s recent adaptation of Great Expectations—stages scenes beautifully and coaxes remarkable work from the cast, but he never finds a way to bring a dynamic touch to the narrative. The film often sits instead of moving, but while its slowness is occasionally frustrating, it also creates an air of free-floating discontent, in which most people seem to accept unhappiness as the norm, and wait to punish anyone who tries to escape it. There’s no happy ending to be wished for, just an occasional thinning of the gloom.