A custom rom is an alternate operating system that can be installed on an Android device. However since every device is different, not all roms will be compatible. Most devices that are shipped by manufacturers such as Samsung and LG come with a skin over regular “vanilla” or stock Android. Some phones however, such as Google’s Nexus and Pixel line of phones run Android without any modifications and are not held down by any bloatware or unnecessary apps. Custom roms allow you to install operating systems based on stock Android or other UIs on devices that don’t ship with it.
To install these new OSes, you need to first unlock your bootloader and flash a custom recovery. To do this on a Nexus/Pixel phone, you can plug your phone into your computer and use CMD or Terminal. On a Samsung Phone, you would use something such as CF-auto root. On HTC/Moto/LG, you can often obtain an unlock code/method from the manufacturer to do it.
After you unlock your bootloader, the next step would be to find a custom recovery supported by your specific phone. The most popular is TWRP (Team Win Recovery Project) and can be downloaded for many different devices at twrp.me. Once it is downloaded, you can use CMD/Terminal or Odin for Samsung to flash it.
Finally, you will be able to browse websites such as XDA for roms and other operating systems built for your device. Once you pick one, download it and find a corresponding Google apps package. Now plug your phone into your computer and copy it over.
Shutdown your phone and hold the Volume Down and Power at the same time, and TWRP should boot. Here you should click backup to create a Nandroid in case anything goes wrong. Transfer this backup to your computer.
Next, click Wipe and swipe the arrow, which will delete all your data, so be careful. Then click install and choose the rom zip file you downloaded earlier. Once this is done flashing, you can install the Gapps zip.
Now reboot your device and wait 5-10 minutes for the first boot. If you did everything properly, you should be able to setup your device and use it!
For more specific steps, search the internet and the XDA forums for detailed instructions pertaining to your device. If anything fails in the process, you can always restore your nandroid to return your device to its previous state.