For games with deeper, more involved rulesets, it can be difficult to teach new players the intricacies of how the various rules interact, stack, offset, or play with one another. A method that I’ve used to teach new players (either new to the system or new to role playing in general), is to railroad them through the rules.
It’s generally accepted that railroading the plot is poor form for a GM. This is because the choices the players make don’t affect the flow of the story. This robs them of their agency, and this thievery of agency can greatly reduce the fun the players are going to have at the table.
However, in the case of presenting rules, you can “railroad” scenarios to present small subsets of the rules in bite-sized chunks for the players to ingest. The approach is to keep the plotline linear for a brief period of time in order to have encounters that build their knowledge of the game. Once you feel they have the basics (plus maybe a few other extras appropriate for their character), then allow them the freedom of choice for left/right/up/down or whatever direction they want to go.
You can keep the plot linear for a short time by having the entrance to the dungeon be more-or-less a straight line without decision points for which direction to go. This will allow you to build out encounters in order. You know they’ll have encounter A, then B, then C, and so on. This grants you, the GM, control over the “teaching session” that you’re presenting to the players.
Start encounter A with something super simple. I would recommend one minor monster per PC. By minor monster, I’m talking things easy to take on like goblins, kobolds, large rats, etc. Just something to wet their appetite and sharpen their skills for the basics. The basics should include what they have to roll to hit, what damage they do when they hit, where to find their armor class (or equivalent) on their character sheet, and where their hit points (or equivalent) are located. For the first fight, make it take two hits (or one really good one) to kill each monster. Just ignore the monsters’ hit point levels. Two hits on each critter will allow the encounter to be brief, but still long enough to be educational.
Also, don’t use advanced tactics or special powers (like a rat’s inflict disease) from the monsters. Keep it simple. If the PCs happen to stumble into an advanced tactic (like flank or higher ground), then pause the combat long enough to explain the bonus or advantage the PC gets for such an action.
Slowly spoon feed new rules and situations to the players as they move their characters through each encounter.
Make The Encounters Interesting
Just because the encounters are simple, that doesn’t remove the requirement of keeping them interesting. Be evocative with your descriptions (use as many senses as you can beyond just “you see”). Make the monsters or things they encounter neat and cool. Drop those powerful descriptions into your narrative to build the world in small chunks. The new players will eat that stuff up. I guarantee it.
Also, think beyond just combat. Add in a room or hallway with a trap. Add in an environmental effect to a couple of the encounters. I’m thinking things like a slippery floor to make those dexterity checks more difficult or a strange fog to impede vision would be nice additions. Perhaps the players will find a way to work these environmental effects to their advantage if they are clever enough.
Lastly, add some social interaction. Most “railroad the rules” scenarios are going to be dungeon-crawl style operations because those offer the GM the most control over the direction the PCs go. That’s fine, but throw in someone that is maybe a prisoner for the PCs to rescue and interact with. Perhaps a disgruntled guard has turned coat against their master and wants to help the characters. This will illustrate to the players that not everything in the dungeon is there to be met with the edge of a sword or point of a spear.
Once you’ve introduced at least half (if not more) of the special conditions, maneuvers, attack modifiers, and so on, hand the players a cheat sheet. The cheat sheet should be no more than a single sheet of paper with print on both sides. The key reason I don’t give them the cheat sheet out of the gate is that I want to control (remember: we’re on a railroad) the pace and ordering of when I introduce the different rules. If I hand the cheat sheet to the uninitiated, it might overwhelm them, confuse them, lead to analysis paralysis, or lead them to break off the railroad of rules learning and get more advanced than you (or the other players) are ready for.
If you have time, you can even create a secondary cheat sheet that is specifically crafted for each character. This is quite a bit of work, so I don’t recommend it. What I do recommend is providing guidance to the players on where to find the rules specific to their character and encouraging them to make a cheat sheet of their own. This will drive home the rules they need for their character and make them more concrete in their own minds.
Again, don’t stay on this railroad too long. You can introduce two to three new effects and rules in each encounter after the first. You’ll be pleasantly surprised at how quickly the players will pick up the rules without having them ingest hundreds of pages of a core rulebook.