Yeah there isn't really a definition for real Doctor Who and that's shown by everyone having a different definition.
There is though, that's what's annoying.
In Classic Who, regeneration was like an advanced form of healing. His body broke down, it repaired itself by rebuilding itself from scratch, giving him a new appearance. His outer personality was also given a shake up as a result, but his consciousness, core persona, memories etc all remained unchanged. The differences between the different Doctors were more akin to the differences we all go through at different stages in our life, only more rapid. He did not become a totally different person each time.
The Doctors core character that ran throughout all of the first 7 meanwhile is as follows.
He is mysterious about his origins and never lets us know his real name.
He is a pompous, aristocratic, somewhat stuffy very British hero (even if he is Gallifreyan LOL I mean in character.) That helps him stand out from other male heroes, and it ties into the characters literary influences as being a Holmesian, gentlemanly adventurer type.
He wants to travel the universe to learn about its people, cultures, etc. He doesn't set out to be a hero, but his strong sense of morality and his curiosity get him into trouble where he is forced to act.
He wants to live life on his own terms and is very much a hedonist. Far from being a hero ironically if he's asked to help he'll often be reluctant as then it becomes work. We see this with Hartnell in Keys of Marinus, Pertwee when he is exiled to earth, Tom in Key to Time etc who all have to be forced, even threatened into helping to save the day.
He is an older, in some ways more mature character who doesn't fear death, can cope with great loss, not let his emotions cloud his brain (unless it is his curiosity) and often serves as a mentor, grandfather figure to his companions. Conversely he can also be childish when it comes to getting his way. Still he is very much a stiff upper lip British hero despite his childishness. We never see him cry for instance, apart from when the regeneration has gone wrong and he is meant to be insane in Twin Dilemma.
He is largely asexual. Though he did have a grand daughter, for whatever reason he isn't interested in romance any more, and his companions often serve as a surrogate grand daughter figure, are too young for him anyway and are of a different species LOL.
He is a Jack of all trades master of none, who has through a combination of his long life, small attention span and time machine that can allow him to visit all the greats, picked up skills in many areas from fighting to medicine.
His moral code is very strict. He will kill his enemies if need be, and using whatever method he has like guns, explosives, bio weapons etc. However if possible he will try and find alternative solutions and he doesn't like killing his enemies when they are unarmed and helpless. However when it comes to recurring foes like Davros, the Daleks and the Master he can sometimes be pushed into killing them if need be, even when they are helpless.
Even physically and in terms of small quirks there are patterns that run throughout the Doctors. They all tend to dress in Edwardian/Victorian era clothing, they all tend wear frock coats, and other motifs like scarfs, hats, and cloaks run through the Doctors costumes, they all tend to have big, long, crazy hair to reflect the characters bohemian nature etc.
Hell even a love of Jelly babies runs throughout multiple Doctors.
Also finally the type of actor they got to play the Doctor in the classic era was usually an older, more eccentric, offbeat character actor, who by that stage at least was firmly typecast in larger than life, supporting roles, such as villains, comic reliefs etc. Hartnell was primarily a villain actor at that point and was even known as Britain's leading menace, Troughton though a leading man when he was younger, was now typecast in offbeat supporting roles on film and tv like in Jason and the Argonauts, Jon Pertwee was a rubber faced comedian, Tom Baker had similarly been a Tim Curry type actor, known for playing mad, bad and dangerous to know villains, Colin similarly though a leading man in his youth, was firmly a bad guy actor at that point, and McCoy, well he has always been the definition of a quirky oddball actor.
Peter Davison was really the one exception, and he was good, which shows you can sometimes go beyond the template in terms of the type of actor you cast, but ultimately there was a particular type of actor in the role of the Doctor.
You are wrong that there is no similarity between Hartnell and Troughton. The myth comes from the fact that you are right that Hartnell had a character arc, where we saw him go from a selfish ass to the Doctor we know and love. Sadly however Hartnell's third season that shows the end result of that is destroyed basically. There's like 2 or 3 stories that survive from it. Not as many people are willing to check out the recons, understandably as they require about 10 times the attention, and on top of that they are of older shows that don't match modern tastes too. Still Hartnell sets the template for who the Doctor is (no pun intended) in season 3.
I am not opposed to the Doctor going through a character arc meanwhile, but the changes have to be natural and flow realisitically from the previous incarnation. Regeneration can only really account for the outer personality changing, which is not nothing, but you can't change the fundamentals for that reason.
Meanwhile the show itself whilst going through multiple different genres, always had a focus on adventure, derring do, and really post Hartnell sci fi of some kind. Also the villains that recurred also had a fairly consistent characterisation and backstory too.
If you wanted to do a sequel to classic who, you had to follow this characterisation and develop from there naturally, and when necessary refer to past continuity, if you were doing a story with an old villain.
If you don't want to do that, fine. Do a remake. I think in a remake you can be more experimental, in fact I think you should. Though even then you still have to stick to the basic template of the character, IE the Doctor still has to be a time traveller, still has to have a blue box, some of his iconic foes should appear etc. Still remakes obviously can play around with things.
However RTD wanted to have his cake and eat it too. He basically wrote it as a remake in the 9th Doctors era, but threw in references to classic era to try and win round its huge fanbase and play on the nostalgia the general public had for it.
As a result we got the following.
The 9th Doctor had none of the Doctors core characterisation.
He doesn't travel because he wants to see the universe. He travels because his people are gone, and he misses his people and just wants to settle down somewhere and have a normal life.
He falls in love with his 19 year old companion.
He dresses in completely modern contemporary clothes , has short hair and looks otherwise normal.
His moral code is all over the place, he kills people for vengeance and anger, yet refuses under any circumstances to use a gun, apart from when he is meant to be mad.
He always lets his emotions cloud his brain, is completely hysterical and loses it several times and sheds tears regularly.
He isn't as much of a Jack of All Trades. We don't see him fighting that much for instance.
Also the 9th Doctor wasn't played by a weird character actor. Eccelston is more what you would call a conventional, unconventional leading man who had played a number of leading men in more conventional dramas. Whilst again he could have overcome that like Davison, he was written as a more conventional, romantic hero.
Eccelston's Doctor was written as being a working class character with a down to earth accent. Again I myself am working class, so it's not like I am saying "oh never have a working class hero" LOL. However the Doctor again is the gentleman adventurer type. Also I am not saying don't cast a working class actor. Hartnell was more working class than Eccelston, and spoke in a working class accent in real life. (Listen to his desert island discs interview.) However he deliberately perfected that posh voice he spoke in as the Doctor. Eccelston didn't bother.
You could have maybe justified some of these changes or worked them in naturally, like his people being gone makes him a bit more unpredictable, but all of them at once was too big a departure, and on top of that, the format of the show changed.
Now just as much focus was put on the companion, if not more, and the show became all about romance, soap opera, love triangles and so on.
All story arcs from classic who, Dalek civil war, Cybermen's story arc from Tenth Planet to Silver Nemesis, the Master's relationship with the Doctor were all tossed in the bin. The Cybermen now had a completely different origin, the Master also had a completely different origin and a completely different pseudo romantic relationship with the Doctor.
These problems only got worse with Tennant, and were not helped by the odd member berry like Sarah Jane (who was also written out of character, by being retconned to being in love with the Doctor.)
Admittedly things were pulled back with Matt. Whilst I think they cast him because he was young and cute, he was a more offbeat kind of an actor, and unlike Tennant and Eccelston who didn't really know anything about the original, he went back and watched it and got to know the character. Also at that point I think Moffat wanted to try and recapture the classic era feel a bit more.
Incidentally it's worth mentioning that Matt Smith is by far the most popular doctor among classic who fans from the revival. Of course that's not to say he was perfect. He did still have a lot of New Whoisms too, but overall he I think bridged the gap between classic and new who. He was more alien and professorial than Tennant and Eccelston, had a more Doctorish moral code, dressed like the Doctor, but he was a bit younger, more vulnerable like the new who Doctors.
On top of that his era removed a lot of the RTD baggage like the Time War, allowing the Doctor to just be a traveller again etc.
Had the Capaldi era been what Moffat intended, an era with darker, more serious sci fi stories, with a more up to date version of the Classic era type of Doctor, then yes. New Who and Old Who could have been compatible in spite of the RTD era, which in hindsight could have just been looked on as a weird period in the Doctors history.
However sadly the new generation of fans that RTD built up in his era, were not happy with it going back to the classic era style. I'm not saying you can't get anyone who enjoys both styles, but overall it's fair to say RTD's core audience didn't much care for classic who. Why would they? It's a totally different type of show to the one they liked.
As Moffat was desperate not to lose them as he felt they were the young audience, he went back the way in Capaldi's time to the RTD era style with a vengeance and that coupled with political climate at the time, caused the Capaldi era to finally cut the revival off from the original. Post Missy there is no going back to it being a proper sequel, and then of course Jodie cements that.
Still you have to ask yourself why did Moffat go back the way? It was because of RTD bringing his new type of fan to the show. PS I'm not saying that type of fan was bad. In fact my best friend in DW fandom, Claudia Boleyn who I absolutely adore, is the epitome of that type of fan. However still that does just show what I'm saying that the revival is a remake. It's like Gary Oldman's Dracula to Christopher Lee's Dracula. Yeah some people who just love the core concept will appreciate both, but ultimately both are made for different audiences.
Sadly however Moff, Chibbers and RTD still didn't want to lose the classic audience either and so they kept crowbarring in all these classic references and characters and again often wrote them out of character too, and worse in order to justify all their changes and retcons, they did stupid stories and came up with lazy explanations like bi regeneration and the Toymaker making a jigsaw out of his history which basically is just "A Wizard did it."
The whole thing is a convoluted mess now as a result, and honestly the only way DW can go on is if the revival finishes and then after a break we get an alternate sequel with a new 9th Doctor. Still have it that new who, along with the Cushing movies and Shalka take place in alternate universes, but otherwise we have to dump gender bending time lords, timeless children, bi regeneration, all this rubbish that just muddies the water.
