Most Mandarin speaker often just think “in fact" = 事實上 / 其實.

So, they will just use “in fact" to add information to show that something is wrong.

-Some politicians claim that the new policy has been a success. In fact/However, it has been a total disaster.

————————————

However, there are two ways to use “in fact."

in (actual) fact

1. used to give extra details about something that has just been mentioned (用在要對之前說的話作補充, 或是舉出例證強調時)*

-I used to live in France; in fact, not far from where you’re going.
-I love his new album! In fact, I love every single song on it!

You can also use “as a matter of fact" when you add information to clarify the previous sentence.

-I’m not a poor worker. As a matter of fact, I’m a very efficient worker!

Sources:
http://idioms.thefreedictionary.com/as+a+matter+of+fact
http://www.bbc.co.uk/worldservice/learningenglish/grammar/learnit/learnitv247.shtml

————————————

2. used to emphasize a statement, especially one that is the opposite of what has just been mentioned (事實上 / 其實)

-I thought the work would be difficult. In actual fact, it’s very easy.

Sources:
http://oaadonline.oxfordlearnersdictionaries.com/dictionary/in+fact#fact__80
http://www.merriam-webster.com/dictionary/in%20fact
http://bbs.pigai.org/t18637-1-1.html
http://tw.myblog.yahoo.com/teacher-sammy/article?mid=643&prev=651&next=-1

———————————————-
actually
1. used in SPEAKING to emphasize a fact or a comment, or that something is really true

-The weather report is so accurate! It’s actually raining now!

2. used to show a contrast between what is true and what somebody believes, and to show surprise about this contrast

-I thought the restaurant was very expensive! However, it’s actually very affordable/cheap!

3. used to correct somebody in a polite way

-We’re not American, actually. We’re Canadian.

Source: http://oald8.oxfordlearnersdictionaries.com/dictionary/actually
———————————————-

as a matter of fact
1. (in fact) used to add a comment on something that you have just said, usually adding something that you think the other person will be interested in

It’s a nice place to stay. In fact, / As a matter of fact, we’ve stayed there ourselves.

2 (actually) used to disagree with something that somebody has just said

A: I suppose you’ll be leaving soon, then?
B: No, as a matter of fact / actually I’ll be staying for another two years.

Source: http://oald8.oxfordlearnersdictionaries.com/dictionary/as+a+matter+of+fact+#matter_1

———————————————-

Difference between in fact, as a matter of fact, actually, well:
1. http://www.bbc.co.uk/worldservice/learningenglish/grammar/learnit/learnitv247.shtml

2. http://tw.myblog.yahoo.com/teacher-sammy/article?mid=643&prev=651&next=-1