In his writings Darby mentions that his mother passed away when he was an infant and did not know her but that he had a portrait of her that he cherished. ??

You'll find more information here:

https://www.brethrenarchive.org/on-the-brethren-trail/grave-hunting/st-mary-churchyard-warbleton/anne-darby/

Darby's words were, "I was early bereft of her" which was assumed by some to mean that she had died when he was young but, the fact is, she died much later when he was about 50. It seems that there had been some issue in the family that resulted in JND's being estranged from her. Perhaps he was whisked off to boarding school and didn't see her (much?) after that?