What the English did to the Irish was unconscionable. The damage has slowed – but it is not undone, and never will be.
Anyway, as you would imagine, much research has gone into determining who owned slaves in the American South and who did not. Many surnames have been listed but, as I have not looked into the matter myself, I can’t say what has been found with regard to the name “Sullivan.”
But @blaneyboy324 did not say there were few Irish slave-holders in the US. He said there were few Irish-Catholic slave-holders in the US.
Relaxing that constraint, or at least ignoring it, yes, you’re obviously right: there were descendants of Irish immigrants who did own slaves. For example: Census records show that, if you climb four generations up the family tree of a certain Mitch McConnell, there are at least two slave-owners there, one of whom was the son of an Irish immigrant, last name McConnell.
As for this:
The judge and his family would likely know, but I don’t.
Plus there’s the matter of his middle name, which is … “Gael”!