I'll try my best to explain in text...
r is a cartesian vector, x i + y j + z k
so dr is dx i + dy j + dz k.
You have to break up your graph into OA, AC,OB, BC, and OC and then add them. that is, if you want OAC, add the work from OA and AC.
notice OA is only travelling in the x (i) direction, so when you take the dot product of F and dr, you'll only get 2y dx because dr is only x i thus the j component in F drops off (rather its multiplied by 0 j because dr has no j component in OA).
So then we integrate, but notice on OA y=0, so that integral is from 0 to 5 of 2(0) dx which is 0, in other words no work is done on OA.
Notice AC is only travelling in the y (j) direction, so you will only have x2 dy in your integrand. this time x = 5 so you will integrate from 0 to 5 of 25 dy which gives 25y from 0 to 5, plug in 25(5) - 25(0) and you get 125, thus the work done on AC is 125 Joules. So now we add OA and AC, so that OAC is 0 J + 125 J = 125 J.
Do that for the others.
The tricky one comes with OC, since it is travelling in both i and j. this one you will have to split up into two integrals going from 0 to 5 on both.
When all is said and done you'll notice that Work was not the same for each path, i.e. work is path dependent, so the Force is non conservative.