First, use the right tool for the job.
Any guitar tone is a combination of three things: The player, the guitar, and the amp. Change any one of those factors, the tone changes. Your fingers sound different to my fingers. You want it to sound like a Mesa with a 4x12? Use a Mesa with a 4x12! A big sealed cab sounds different to an open backed combo, end of story. Not to say worse, just different. A Fender Twin delivers best what a Fender Twin delivers. So initially try to use something as close as your goal as possible. I know this is common sense, but you'd be surprised how many people will show up with the wrong thing and have expectations that you will struggle to meet. A single coil equipped Fender Strat is not a firebreathing metal monster off the shelf. But maybe it is with a different pickup, or some kind of pedal...ask Yngwie!
Perception is your enemy in the studio when it comes to guitars, in a big way.
Is it good because it is vintage? No.
Is it good because your hero <insert hero here> used it? No.
Is it good because you heard it was done X way? No.
This list is a very long list. Over all, trust your ears rather than your head. Check tuning religiously. Wanging the living crap out of a whammy bar as a kid, I spent 50% of my time out of tune. Which is hindsight as not helpful. Adding to that a faint smear of stereo chorus filth adding to the tuning weirdness I probably would have slapped my late teenage self when it came to tuning habits.
So let's assume we got our hands on a suitable guitar setup, we are in tune and we are in the studio eager to lay down guitar magic. Here's the first thing: Everything sounds good loud. This does not mean it IS good. Typically you mic up the amp, walk in the control room and go "Oh", followed by a frown.
Getting started with a dynamic
I’m going to assume you already have your rig together; it’s a huge topic which I will cover in another post.
Try to get the best basic tone using one well placed mic. As a starting point, place an SM57 (or 58, Beyer 201, 421, EV or…any other dynamic) approximately 2" to the side of the cone about an inch back from the grill. You can, if you are anal/thorough try all the speakers to see if any of them sounds better than the others. They might.
Starting from a position of face on to the cabinet at 90 degrees, have a listen. Angle it towards the distance cap of the cone for brighter, away from less brightness. Pull it backwards for less bass, closer for more. And that's about it. You can of course EQ, and feel free to. That's what it's for.
Most of whether or not you have something decent at this point is down to your tone and settings. That's why its helpful to be able to change the settings on the amp in the control room, and have a speaker cable running to the live room as the perspective is a lot different once mic's are placed and the sound is coming out of a 5" monitor speaker ay conversation volume!
I have had the most success capturing an in-the-room tone with ribbon mics. Place a Royer R121 dead centre on the cone of the speaker, a hands width back from the grill, tilted slightly down and twisted slightly to the left or right to stop ribbon resonance. In fact - try this positioning with any ribbon - they hate head-on sound waves. Also, with any ribbon, consider a pop shield as LF tends to make things get weird. When you hear this in the control room, especially if you are doing modern rock you will instantly think it is too dull - which it is. That's why we have EQ and why we fearlessly put some knob on it. We are engineering, not setting a gain level! For this task I like the API550 EQ, available in relatively affordable hardware, and of course plugin versions.
Condensers vary a lot on guitars from nice and warm, to too bright, to too...good...for want of a better way to explain it! So for this, try your options. Again for me, this will be my second choice if there's no ribbon and I have access to a decent mic locker. Condensers also get roped in as a second mic along with a dynamic, which can also work, or often just makes things worse.
Place the first mic and get it sounding as good as possible. Turn up the headphones really loud - and do not under any circumstance allow anyone to PLAY anything - and listen to the hiss from the first mic. Pull the second mic back unit the hiss sounds as fat as possible. At this point you stand good chance of being in phase and things sounding bearable - then go from there. For this technique I am still talking about relatively close to the amp.
Some people insist on micing the room, and that comes down to one thing - does the room actually offer anything worth recording? Most small spaces, and most deader live rooms of any size the answer is no; you will just be recording indistinct mush. Sometimes, the room sounds cool, but rarely. For this, when working on more "organic" music I prefer to create the room later using any one of the awesome plugins we now have access to at a click of the mouse as I can control and tweak it to fit, typically a convolution reverb IR of a real space.
Another tip for you - if you combine mics, try busing them to an aux and eq'ing them both at the same time after they are in phase, as this results in less LF phase shift.
These days nobody commits to anything in a DAW, so your elaborate multi miked setup while good intentioned is often a mess. Often when I get multi mic'd tracks I'll just find the best sounding mic and mute the other one. No phase issues, no messing around. I also don't care what it IS - it could be a 57 or an expensive tube mic - which ever sounds best in the context of the track is the one that stays.
What about amp sims? Well, I have lived through the evolution of these things from the Sansamps, to the POD's and Ampfarms, Zooms all the way to the Kempers and Scuffham plugins and the good news is it only gets better and better. I would still like to burn this one thing Korg made, however. Made of plastic, sounded like plastic. There is and will always be a divide between those who believe that an amp sim can compete with an amp, and even as someone who collected amps for years, I am starting to fall on the side of feeling that with a good cabinet IR and sim, you are as likely to get as good a tone this way given most people's self-recording circumstances at least. From at least this is good news as it brings us all to closer to better tone.
It's a pretty good idea to record a DI as well as your mic'd signal, especially if your amp selection is crap, or you happen to be Nile Rogers and you want to record some glassy front pickup Stratocaster funk. The only thing I hate DI on is acoustic guitar especially if it is the only option. There is nothing wrong with using the flexibility we are afforded now to fill in with something later with that which our amp didn't provide. The downside of that is options and choice which are the enemy of creativity.