Answer : 9/16
Let hexagon A have the vertices A1, A2...A6 and B have the vertices
B1...B6. Each of the sets of vertices is numbered clockwise such that A1
lies 'between' B6 and B1, A2 lies 'between' B1 and B2, and so forth.
Consider, for example, the isoceles triangle B1-A2-B2. The angle
B1-A2-B2 is 120 degrees (the interior angle of a regular hexagon). Let
the length of segment B1-A2 be (1/2)x, where x is the side length of
hexagon A. Let the side length of hexagon B (which is the segment B1-B2)
Then, by cosine rule on triangle B1-A2-B2,
y^2 = [(1/2)x]^2 + [(1/2)x]^2 - 2*(1/2)x*(1/2)x*cos120 deg.
cos 120 deg = -1/2
Doing the algebra, we get:
(y/x)^2 = 3/4
Since hexagons A and B are similar, their areas are related by the
square of the ratio of their corresponding dimensions. This is in fact
the expression we've just found (y/x)^2, which is equal to 3/4.
So the ratio of area of hexagon B to A is 3/4. Similarly (pun intended!), the ratio of area of hexagon C to B is also 3/4.
Hence the ratio of area of hexagon C to A is (3/4)*(3/4) = 9/16.