Pair programming is the answer. Let's have a look at your current project state. We will use an informal metric introduced by Jim Coplien - the "truck number" metric. The very essence of this metric is the question "How many or few people would have to be hit by a truck (or quit) before the project is incapacitated?" Obviously, the worst number is "one." How does your project score?
What can you as a manager do about the minimizing of this risk? Well, you may object that you do not want to put two people to work on a job that can be done by one. But the fact is that one of the benefits that pair programming brings is that it spreads the knowledge across the team and that in turn increases the truck number and project safety.