memtest is a program to test if your memory is alright or not
they come in a few different kind of packages
-either an executable that will make a bootable floppy or bootable USB key
-or a CD Image (ISO) which has to be burnt on CD (after being unzipped of course) and will make a bootable CD
in any case you'll end up with a bootable media which you should boot on
windows will be bypassed and the specific software of memtest will be launched and it will then proceed to testing your memory and telling you whether or not it found any problem in your hardware
I advise you to read the website (I know it can be a bother) as it contains a lot of information on how to do this & that and what kind of results to expect and such ...
Lastly, this should test out your RAM but in some cases you can get other errors ... and sometimes it can also be because your computer BIOS is not properly configured...