Let choose any point P on AB. The problem is now to construct an equilateral triangle PQR, Q and R being on two given lines, which is a classical problem :
There are generally two solutions, corresponding to the ±,
excepted if the angle of the two lines is π/3, then D'1 is parallel to D2 : only one solution,
or D'1 coincident with D2 : infinitely many solutions.
In our case, only one solution at most is suitable for Q,R in the sides BC,AC.
All these methods give infinitely many PQR equilateral triangles, with vertices on the sides of a given ABC triangle.
The smallest triangle is then "of course" the one with QR perpendicular to the angle bisector : the circumcircle to PQR goes at C, it will be the smallest, hence also will be PQR, when PC is a diameter in this circle. QR is then the perpendicular at 1/4 of angle bisector from C (or also PQ and PR are perpendicular to BC and CA).
I'll say nothing about the case when ABC itself is equilateral...
The general case is much harder.
Consider triangle TUV parallel to PQR through vertices of ABC and name x = angle APR
Then ARP = π - A - x (sum of angles in APR)
At P : QPB = π - x - π/3 = 2π/3 - x, and in triangle PQB angle PQB = π - B - (2π/3 - x) = π/3 - B + x.
In triangle APR : PR/sin(A) = AP/sin(π - A - x)
In triangle PQB : PQ/sin(B) = BP/sin(π/3 - B + x)
As AP + PB = AB and PQ = PR : AB/PQ = sin(A+x)/sinA + sin(π/3 - B + x)/sin(B)
Consider now the triangles ACV and BCU
Angle CAV = ARP = π - A - x (PR//AV) hence CV/sin(π - A - x) = AC/sin(π/3)
Similarily in BCU : CU/sin(π/3 - B + x) = BC/sin(π/3). Hence UV = AC.sin(A + x)/sin(π/3) + BC.sin(π/3 - B + x)/sin(π/3)
As AC/sin(B) = BC/sin(A) = AB/sin(C) : UV = AB.sin(B)sin(A)/(sin(C)sin(π/3)) × (sin(A + x)/sin(A) + sin(π/3 - B + x)/sin(B)) = AB.sin(B).sin(A)/(sin(C)sin(π/3)) × AB/PQ
A first immediate conclusion is UV.PQ = const Hence :
|PQR minimal ⇔ TUV maximal|
More precisely, Area(ABC) = 1/2 AB.AC sin(A) = 1/2 AB²sin(A)sin(B)/sin(C), Area(TUV) = 1/2 UV²sin(π/3) and Area(PQR) = 1/2 PQ²sin(π/3), hence :
|Area(TUV)/Area(ABC) = Area(ABC)/Area(PQR)|
But now, the problem of finding the maximum TUV has been solved (in erased triangle). Recall the result : TUV maximal iff it is parallel to the "Napoleon triangle" IJK, built by the centroids of equilateral triangles constructed on the sides of ABC, outside ABC, that is the centers of circles from where we see the sides under a π/3 angle, outside ABC. As PQR min // TUV max :
|PQR min ⇔ PQR // IJK|
A simple construction of PQR min could then be :
Construct equilateral triangles outside ABC on the sides AC and BC, and their centroids I and J.
Line IJ, or a parallel line, intersects BC and AC in M and N.
Construct an equilateral triangle HMN in the half plane opposite to C.
Line CH intersects AB in P, parallels from P to HM and HN intersect sides BC and AC in Q and R.
For generalizing to triangles PQR of any shape, see nested triangles where a more general and elegant proof of Area(TUV)/Area(ABC) = Area(ABC)/Area(PQR) can be found.
A relation with isodynamix and isogonal centers.