Yes there is a guy who has square tube(I think or its rectangle tube). His name on here is Flipper.
For the most part, it is self preference per application.
It is round to a straight rectangular long "Chute" with a 80deg turn. This changes the sound drastically. However, to maintain some sort back pressure, My CFM/Flow/Temps/RPM Calcs were done by a local shop called "Speed Unlimited-House of Balance" by some fellow mechanical engineers. I was amazed as to how much mathematics and variables were involved.(They hand make custom exhaust for all kinds of race cars and more) There are small 1/4" baffles welded in the 2x3 Square "chute" portion of my exhaust to make the exhaust more turbulent and slow it down, thus gaining lost back pressure. They nailed it. A good portion of Mason Dixon racers now have my similar exhaust.
Oval will provide similar performance.
Yes, in some cases, round pipe does have better flow characteristics within a smaller volume pipe. However, rectangular pipe, if made correctly, will have very similar performance(if not better), and will typically have larger volume, enabling it to handle higher CFM flows. Take a 2" Circle and a 2" Square, The square has more volume. In theory, a smaller exhaust could be used
if it were rectangular with little bends or resistance. However, I just use the termination section of the exhaust system in rectangular medium.(304 2x3x.120 Stainless Steel)
This is actuality if more self preference- flat or round. At work I also design & Engineer ductwork for HVAC Air flow systems.
Some days I use rectangular, some days I use round spiral...this depends on where it needs to be. Exposed I use round spiral duct, for applications where it needs to be hidden in a drop ceiling, I use flat.
One reason I started using use rectangular exhaust(tail piece) 9 years ago, is no one else does or did, and gets it right. Why be a clone?.
The higher register of frequency has a shorter, weaker wavelength, when it hits a flat surface it weakens significantly.
It does not deteriorate as much in round pipe.
The lower register of frequency has a stronger, longer wavelength, thus making it thru the pipe with little deterioration.
This will give the lower register "Note" (Sound) of something aggressive & more desirable, not so much a noisy rattle trap.
The old philosophy of cutting the pipe at 18" or where the paint burns off is exactly that...a far fetched philosophy that someone made up for one particular application and engine, not for all.
FYI.....key in "Nascar Exhaust" under google images........As mentioned above by Birdman, NASCAR uses rectangular chutes more often than not, and there is a reason for that. Space restraints due to low ground clearances, They need to maintain the flow volume so the obvious choice was a rectangular chute. At the time, the unknown benefit was a surprisingly different tone or note. Not really proven if any gain in performance.