To use the Wikipedia's definition, Impact Factor is "a measure reflecting the average number of citations to articles published in science and social science journals." This innocent definition hides underneath it a world of problems, which means the IF should be used with caution. For example, many journals publish non- research papers (commentaries, opinions, etc.). If those get cited, the citation is added to the journal's citation count, but the number of papers remains the same. So, journals who don't publish this sort of articles are at disadvantage in comparison with those who do.
Even in a high-impact journal, papers don't necessarily have the same quality. So, being published in a high-impact journal doesn't automatically mean the paper has a larger impact than those in the less known journals.
The quality of a paper isn't the only factor influencing its citing. Papers in Open Access journals get, on average, more frequently cited than those behind a pay wall.
There is also the time factor - the IF is generally calculated for the two previous years. That means that if a paper is a "late bloomer" and only get cited after that time period, those citations won't be represented in the publishing journal's IF.
A good introduction to the subject is this paper (published in Retrovirology, of all places).