How does religion stop Hamlet from taking his revenge in Hamlet?

In Shakespeare's Hamlet, Hamlet
believes Claudius is doing more than just praying when he comes upon him in Act 3.3:  he
thinks he is confessing. 

The play features Catholic
theology more than it features Protestant.  The Ghost wandering around until his sins
are burnt and purged away is like the Catholic belief in purgatory, not like anything
Protestants believe in:

readability="16">

I am thy father's
spirit,

Doomed for a certain term to walk the
night,

And for the day confined to fast in
fires,

Till the foul crimes done in my days of
nature

Are burnt and purged away....  (Act
1.5:9-13)

This is purgatory,
or something close to it. 

Applying Catholic theology to
Hamlet's refusal to kill Claudius at prayer, then, suggests that Hamlet is reacting to
the belief that if Claudius is confessing, his soul would already be purged and he would
be guiltless before God and would therefore go straight to
heaven. 

It is no accident that Claudius in his prayer
stresses forgiveness. 

readability="32">

...Whereto serves
mercy

But to confront the visage of offence [what is mercy
good for if not to meet sin face-to-face]?

And what's in
prayer but this twofold force,

To be forestalled ere we
come to fall,

Or pardoned being down [what is prayer for if
not to stop us from sinning, or to forgive us when we do]?    Then I'll look
up.

My fault is past.  But, O, what form of
prayer

Can serve my turn?  'Forgive me my foul
murder'? 

That cannot be, since I am still
possessed

Of those effects for which I did the
murder--

My crown, mine own ambiton, and my
queen.

May one be pardoned and retain th' offence?  (Act
3.3.36-56)

Claudius stresses
confession and mercy and forgiveness, and that's what Hamlet
suspects.

The problem with Hamlet's decision to not kill
Claudius because he doesn't want to send him to heaven, is that that places Hamlet in
the role of God.  In other words, Hamlet is playing God, here.  He is messing where he
doesn't belong.  And the consequence for his actions is the blood bath at the end of the
play.  This is probably the climax of the play, the point at which Hamlet and so many
others are doomed. 

Ironically, Claudius does not confess. 
He cannot.  He is at least intellectually honest enough to realize that he cannot be
forgiven when he hasn't repented, when he hasn't given up that which he got by sinning
in the first place.  Hamlet could have revenged his father by killing Claudius, and not
sent him to heaven.