Due to bug in StratixV® ES silicon, Gen2 PCIe® Hard IP fails to link up in Gen3 slot. This issue is affected in Quartus® 11.1sp1 and later versions.
There are two ways to workaround the problem
1. Modify the BIOS of RP to be capable up to Gen2 speed. This means the slot can support either Gen1 or Gen2 only. Using this setting, the link will train up to Gen2 with the Gen2 HIP configuration.
2. If BIOS option is not available for RP, regenerate the core to support maximum Gen1. With this configuration, the link will come up to Gen1 speed.
This issue is already fixed in all StratixV® production silicon.