Some of Mat's memories are his own from either past lives/life or from his bloodline (Manetheren King Aemon evidenced upon waking from his cleansing in the White Tower in book 3). I have always believed he is Aemon reborn but this was never confirmed. The only confirmed fact is that Mat is of Aemon's bloodline. The rest of his memories were given to him by the Finns in book 4. Those memories are not his own lives but of 1000's of men spanning a specific 1500ish years. This is evidenced at one point when he recalls the same battle/war from 3 different men.