I had Q-rings on my P3C and I got them to work flawlessly with the original derailleur setup. You will have to adjust you derailleur angle and height since you switched from circle rings. Maybe mess with the limit screws as well..... It may be best to start from scratch with instructions from Calvin.
https://www.parktool.com/...erailleur-adjustment

Did you raise the front derailleur?

I did. I went from 52T round rings to 52T Q-rings. The derailleur is about 1mm above the outer Q-ring at the largest radius part.

I have a 105 5800 11 speed derailleur and chain.

I had a 52/39 setup when I had no issues. You probably have a smaller small chain ring and a shim may be what you need. Ebay has a bunch




https://www.ebay.com/itm/Folding-Road-TT-Bike-Front-Derailleur-Spacer-Shim-Kit-Slanted-Adjuster-Washer/262952871431?_trkparms=aid%3D1110006%26algo%3DHOMESPLICE.SIM%26ao%3D1%26asc%3D20200520130048%26meid%3Da20323add8024973838762dfda01168d%26pid%3D100005%26rk%3D5%26rkt%3D12%26mehot%3Dpf%26sd%3D324117028364%26itm%3D262952871431%26pmt%3D1%26noa%3D0%26pg%3D2047675%26algv%3DSimplAMLv5PairwiseWebWithDarwoV3BBEV2b&_trksid=p2047675.c100005.m1851

I have two bikes with Q-rings. My 2014 P3 requires no shims on the derailleur and works fine. My road bike needed a ~2mm untapered spacer.

I think the goal when setting it up is to check that the gap from the bing ring at its largest diameter to the outer derailleur plate is constant as you rotate the crank. If the gap is smaller when the peak is at the rear of the derailleur than at the front side, then you’ll likely need a shim of about the difference in this gap. If the gap is larger in front, you may need a tapered/angled shim to tilt the derailleur downwards in the rear. Once you achieve a constant gap slightly larger than 1mm, shifting should be about as good as with a round ring. I made my own shim with cheap aluminum stock, a dremel tool and a file.