Dad JOSEPH WHEATON had no problem letting somebody else raise his 1-year-old son right after his wife died. After the child was around six or seven, Dad got the whim to take the child back, but then would lose interest and return him once more. On the last occasion, the little boy said he wanted to go back to the only home he had known, so Dad beat him to death.
Note that in the 19th century, fathers seldom expressed any interest in raising young children and usually farmed them out elsewhere when the mothers died. In fact, after my great-grandmother died in childbirth, my great-great grandparents raised my grandmother. She seldom saw her father while growing up, and he seemed to be fine with that. This scenario was not uncommon. Nowadays, we expect dads to do the caretaking while the mothers work or to assume "equal" caretaking upon a divorce or separation--often with disastrous results.
Also note that 19 century communities were outraged by violence against a young child, and often took up arms against the killer. A big contrast to the apathy you see today.
From the archives of the New York Times, February 3, 1871:
A Child Whipped to Death by His Father in Indiana--The County in Pursuit of a Murderer
DETROIT, Feb. 2.--Information has just reached this city of a murder, which took place on Monday last, in the township of Wayland, Alleghany County, Mich. JOSEPH WHEATON, a notorious and desparate character, lost his wife about six years ago. They had a son about one year old, whom WHEATON gave to a Mr. BAIRD, who kept the child without interference until last Fall. Since then WHEATON three times took possession of the child, and on Monday last, because the boy wished to return to Mr. BAIRD'S family, he was whipped to death. The excitement in the neighborhood is very great, and the people have armed themselves and are hunting for the murderer.