Let [tex]P=(x,y,z)[/tex] be an arbitrary point on the surface. The distance between [tex]P[/tex] and the given point [tex](7,11,0)[/tex] is given by the function
[tex]d(x,y,z)=\sqrt{(x-7)^2+(y-11)^2+z^2}[/tex]
Note that [tex]f(x)[/tex] and [tex]f(x)^2[/tex] attain their extrema, if they have any, at the same values of [tex]x[/tex]. This allows us to consider the modified distance function,
[tex]d^*(x,y,z)=(x-7)^2+(y-11)^2+z^2[/tex]
So now you're minimizing [tex]d^*(x,y,z)[/tex] subject to the constraint [tex]z^2=xy[/tex]. This is a perfect candidate for applying the method of Lagrange multipliers.
The Lagrangian in this case would be
[tex]\mathcal L(x,y,z,\lambda)=d^*(x,y,z)+\lambda(z^2-xy)[/tex]
which has partial derivatives
[tex]\begin{cases}\dfrac{\mathrm d\mathcal L}{\mathrm dx}=2(x-7)-\lambda y\\\\\dfrac{\mathrm d\mathcal L}{\mathrm dy}=2(y-11)-\lambda x\\\\\dfrac{\mathrm d\mathcal L}{\mathrm dz}=2z+2\lambda z\\\\\dfrac{\mathrm d\mathcal L}{\mathrm d\lambda}=z^2-xy\end{cases}[/tex]
Setting all four equation equal to 0, you find from the third equation that either [tex]z=0[/tex] or [tex]\lambda=-1[/tex]. In the first case, you arrive at a possible critical point of [tex](0,0,0)[/tex]. In the second, plugging [tex]\lambda=-1[/tex] into the first two equations gives
[tex]\begin{cases}2(x-7)+y=0\\2(y-11)+x=0\end{cases}\implies\begin{cases}2x+y=14\\x+2y=22\end{cases}\implies x=2,y=10[/tex]
and plugging these into the last equation gives
[tex]z^2=20\implies z=\pm\sqrt{20}=\pm2\sqrt5[/tex]
So you have three potential points to check: [tex](0,0,0)[/tex], [tex](2,10,2\sqrt5)[/tex], and [tex](2,10,-2\sqrt5)[/tex]. Evaluating either distance function (I use [tex]d^*[/tex]), you find that
[tex]d^*(0,0,0)=170[/tex]
[tex]d^*(2,10,2\sqrt5)=46[/tex]
[tex]d^*(2,10,-2\sqrt5)=46[/tex]
So the two points on the surface [tex]z^2=xy[/tex] closest to the point [tex](7,11,0)[/tex] are [tex](2,10,\pm2\sqrt5)[/tex].
