Target __imp__’s GeoMe

Platform Windows
Language x86 assembly
My rating 2/10
Tools used PEiD, IDA Pro
Solution made by Christopher ’Trundle’ Schmidt
Date November 4, 2007

1 Getting started
We execute GeoMe.exe to get a glimpse of the crackme. It looks like a regular native
window with ownerdrawn content. By pressing the left mouse button a vertex is added
to the canvas. If there are already other vertices the new vertex will be connected with
the previous one. By pressing the right mouse button the drawn polygon is validated
and there is a good boy/bad boy message for valid/invalid polygons.

PEiD only reports MASM32 / TASM32 so we can load the application straight in
IDA Pro. Following the flow at the entry point we immediately recognize the struc-
ture of a native windows appliction. A window class is registered (.text:0040109C),
a window is created (.text:004010E8) and all window messages are handled
(.text:00401102). Once WM_QUIT is received the application returns. Hence the main
crackme functions are located in the WndProc of the main window (.text:00401130).

2 Inside WndProc
The coding of the WndProc is pretty clean. WM_CREATE (.text:0040113),
WM_DESTROY (.text:0040114E), WM_COMMAND (.text:0040116B), WM_LBUTTONDOWN
(.text:004011A3), WM_RBUTTONDOWN (.text:00401274), WM_SIZE (.text:00401335)
and WM_PAINT (.text:004013E2) are handled.

In the following subsections we will have a deeper look at the four most important
messages (WM_PAINT, WM_SIZE, WM_LBUTTONDOWN, WM_RBUTTONDOWN) and how they are
handled.

2.1 WM_PAINT
1 shows the most important part of the WM_PAINT handler. The number loaded in ecx
contains the number of the vertices of the drawn polygon. That is why I name this
memory location dwNumVertices. The location .data:0040341C contains the vertices
itself, stored as POINT’s. I name this location aVertices.
2.2 WM_SIZE
2 shows the most important part of the WM_SIZE handler. The loword of the lParam
(the new width) and the hiword of the lParam (the new height) are loaded into eax and
ecx. Next, their product is divided with four, stored (I name this memory location
dwWindowSize), divided with two and stored again (I name this memory location
dwWindowSize_2).

The remaining code just stretches the x- and y-attributes of the vertices in
aVertices according to the new window size. There is no need for me to explain
this process as we do not need to understand it for solving the crackme.

2.3 WM_LBUTTONDOWN
3 shows the most important part of the WM_LBUTTONDOWN handler. If there are 32
vertices, a WM_RBUTTONDOWN message will be sent, otherwise the vertex will be added.
We now know that there are 32 vertices at most, so we set the number of elements of
aVertices to 32.

2.4 WM_RBUTTONDOWN
4 shows the most important parts of the WM_RBUTTONDOWN handler. If dwNumVertices
is below or equal to 2 we will get the BadBoyMessage.

The following code connects the last vertex we added to list with the first one (a
simple LineTo call).

Finally, the application ends up in calling a function (I name this one Check). If the
return value (__stdcall) is not 0 then the wanted good boy message will be displayed.
In a real life application, if we wanted to crack this application, it might be enough
to patch the jnz with a jmp, but, as we are interested in the algorithm itself, we will
have a deeper look at Check and check how deep the rabbit-hole goes...

3 Inside Check
Check starts right away with a call to another function. As this function updates
various internal values I name this function Update.

3.1 Inside Update

The first part of this function (.text:00401507-.text:00401544) searches the vertex
which has the lowest y-value. If there are several vertices which all have the same (the
lowest) y-value, the vertex with the highest x-value will be taken. The index of the
found vertex is stored in edx.
 Next (5) an array is filled with the numbers from 0 to dwNumVertices-1. It is
obvious that this array contains indices, so I name it adwIndices. The number of
elements is the number of elements of aPoints - 32. After filling the array with the
indices the first index (0) is swapped with the index at edx (index of the topmost
vertex).

Then dwNumVertices-1 is loaded into edx and &adwIndices[1] is loaded into ebx.
The function at .text:004015E1 is called. I name this function SortAdwIndices
as it sorts the indices array in ebx in clockwise order (7). This function inter-
nally uses the function at .text:004016A2. I name this one CompareVertices
(6). This functions determines whether two vertices (specified by their indices) and
aPoints[adwIndices[0]] are clockwise by checking the sign of the two dimensional
cross product (right-hand rule).

There is one slight problem with SortAdwIndices. How would the function
go with the vertices in 8? Yes, there is no real way to order it fully clockwise
(1). The next part of Update (.text:00401577-.text:004015DF - 10) detects
those anticlockwise vertices in the SortAdwIndices array and saves the valid ones
in the array at the memory location at .data:004032EE and the number of ele-
ments in the memory location .data:004032EA. I call the array adwValidIndices
(its size must be equal to the size of adwIndices) and the memory location for
the number of elements dwNumValidIndices. Internally the new ordering is pretty
easy. Again the cross product is used (check the function at .text:00401643 -
9). The first two vertices are pushed on the stack and two counter (ecx and
dwNumValidIndices) are set to 2. Then every vertex (aPoints[adwIndices[2]] ...
aPoints[adwIndices[dwNumVertices-1]]) is checked whether it is clockwise with
the two vertices whose indices are on the top of the stack. If they are clockwise, the
index of the checked vertex will be pushed on the stack and the counters will be in-
creased. If the vertices are not clockwise, the vertex on top of the stack will be popped
and the loop will be executed again without any changes to the counters. Finally, once
ecx reaches dwNumVertices, ecx vertices are popped back in adwValidIndices.

The algorithm implemented in Update is referred to as Graham Scan. This

algorithm computes the convex hull of a specific set of vertices. That means,
adwValidIndices stores the indices of the vertices that inclose all vertices in
aVertices.

3.2 The final validation...

3.2.1 The first requirement
The first requirement for a valid polygon can be seen in 11.
(dwNumVertices*(dwNumVertices-1)+2)&7 must be equal to dwNumValidIndices.
3.2.2 The second requirement
The second requirement for a valid polygon can be seen in 12 and 13. The surface
area of the polygon specified by the vertices of adwValidIndices must be between
dwWindowSize_2 and dwWindowSize.

4 Drawing a valid polygon

As there are only these two requirements, drawing a valid polygon is simple. To meet
the first requirement we just have to draw a polygon whose number of vertices that
inclose all vertices (dwNumValidVertices), combined with the total number of vertices
(dwNumVertices), is valid (2).

Meeting the second requirement is simple as well. We just have to draw our polygon
so big or small that the surfae area of the convex hull ranges of our polygon between
dwWindowSize_2 and dwWindowSize.

14 shows a valid polygon. The first vertex is the left one. There are seven vertices
(dwNumVertices==7), four of them specify the convex hull (dwNumValidVertices==4).
The content of adwIndices can be seen in 3, the content of adwValidIndices can be
seen in 4.

5 Conclusion
This crackme is definitely a simple one. There is no anti-debugger code, the code is not
obfuscated and the algorithms used are well-known and simple to identify. Nevertheless
this crackme is one of the unusual crackmes and solving it was fun. I certainly hope
you find this article a comprehensive and useful source of information.
.text:004013F7 mov ecx, dwNumVertices
.text:004013FD test ecx, ecx
.text:004013FF jz short loc_40142A
.text:00401401 cmp ecx, 1
.text:00401404 jz short loc_401414
.text:00401406 push ecx ; int
.text:00401407 push offset aVertices ; POINT *
.text:0040140C push [ebp+hDC] ; HDC
.text:0040140F call Polyline;
.text:00401414
.text:00401414 loc_401414: ; CODE XREF: WndProc+2D4
.text:00401414 push 0 ; COLORREF
.text:00401416 push aVertices.y ; int
.text:0040141C push aVertices.x ; int
.text:00401422 push [ebp+hDC] ; HDC
.text:00401425 call SetPixel

Figure 1: WM_PAINT

.text:00401340 mov eax, [ebp+lParam]

.text:00401343 and eax, 0FFFFh
.text:00401348 mov ebx, [ebp+lParam]
.text:0040134B shr ebx, 10h
.text:0040134E mul ebx
.text:00401350 shr eax, 2
.text:00401353 mov dwWindowSize, eax ; Width*Height/4
.text:00401358 shr eax, 1
.text:0040135A mov dwWindowSize_2, eax ; Width*Height/8

Figure 2: WM_SIZE
.text:004011B1 cmp dwNumVertices, 1Fh
.text:004011B8 jbe short loc_4011D0
.text:004011BA push 0 ; lParam
.text:004011BC push 0 ; wParam
.text:004011BE push 204h ; Msg
.text:004011C3 push [ebp+hWnd] ; hWnd
.text:004011C6 call PostMessageA

Figure 3: WM_LBUTTONDOWN

.text:0040128A mov [ebp+hDC], eax

.text:0040128D mov ebx, dwNumVertices
.text:00401293 cmp ebx, 2
.text:00401296 jbe short BadBoyMessage

.text:004012C5 call Check

.text:004012CA test eax, eax
.text:004012CC jnz short GoodBoyMessage

Figure 4: WM_RBUTTONDOWN

.text:00401544 mov ecx, dwNumVertices

.text:0040154A
.text:0040154A loc_40154A: ; CODE XREF: Update+51
.text:0040154A dec ecx
.text:0040154B mov adwIndices[ecx*4], ecx
.text:00401552 jnz short loc_40154A
.text:00401554 mov adwIndices[edx*4], 0
.text:0040155F mov adwIndices, edx

Figure 5: Update: .text:00401544-.text:00401565

.text:004016A2 ; on return: eax==1: clockwise
.text:004016A2 ; eax==0: cross product=0 & distance of IndexA and IndexB
.text:004016A2 ; to adwIndices[0] is equal
.text:004016A2 ; eax==-1: anticlockwise
.text:004016A2 ; Attributes: bp-based frame
.text:004016A2
.text:004016A2 ; int __stdcall CompareVertices(int dwIndexA,int dwIndexB)
.text:004016A2 CompareVertices proc near ; CODE XREF: SortAdwIndices+10
.text:004016A2 ; SortAdwIndices+29
.text:004016A2
.text:004016A2 dwIndexA = dword ptr 8
.text:004016A2 dwIndexB = dword ptr 0Ch
.text:004016A2
.text:004016A2 push ebp
.text:004016A3 mov ebp, esp

; (aPoints[IndexA].x-aPoints[adwIndices[0]].x)*;
; (aPoints[IndexB].y-aPoints[adwIndices[0]].y)
.text:004016A5 mov esi, [ebp+dwIndexA]
.text:004016A8 mov edi, [ebp+dwIndexB]
.text:004016AB mov ebx, adwIndices
.text:004016B1 mov eax, aVertices.x[esi*8]
.text:004016B8 sub eax, aVertices.x[ebx*8]
.text:004016BF mov edx, aVertices.y[edi*8]
.text:004016C6 sub edx, aVertices.y[ebx*8]
.text:004016CD imul eax, edx

; (aPoints[IndexB].x-aPoints[adwIndices[0]].x)*
; (aPoints[IndexA].y-aPoints[adwIndices[0]].y)
.text:004016D0 mov ecx, eax
.text:004016D2 mov eax, aVertices.x[edi*8]
.text:004016D9 sub eax, aVertices.x[ebx*8]
.text:004016E0 mov edx, aVertices.y[esi*8]
.text:004016E7 sub edx, aVertices.y[ebx*8]
.text:004016EE imul eax, edx

.text:004016F1 sub ecx, eax

.text:004016F3 jz short CrossProductZero ; compare using
.text:004016F3 ; pythagoras
.text:004016F5 js short CrossProductSigned
;Cross product not signed...

Figure 6: CompareVertices
void SortAdwIndices(DWORD* pdwIndices, DWORD dwNumElementsToSort)
{
if(dwNumElementsToSort<=1)
{
return;
}

DWORD dw=dwNumElementsToSort-1;
DWORD dw2=0;
for(;;)
{
//is clockwise?
if(CompareVertices(pdwIndices[0], pdwIndices[dw])>0)
{
--dw;
continue;
}

for(;;)
{
if(dw==dw2)
{
std::swap(pdwIndices[0], pdwIndices[dw]);
SortAdwIndices(pdwIndices, dw);
SortAdwIndices(&pdwIndices[dw+1], dwNumElementsToSort-(dw+1));
return;
}

++dw2;
//is anticlockwise?
if(CompareVertices(pdwIndices[0], pdwIndices[dw2])<=0)
{
std::swap(pdwIndices[dw], pdwIndices[dw2]);
break;
}
}
}
}

Figure 7: C++ implementation of SortAdwIndices

 Figure 8: How would SortAdwIndices go with these vertices?

int __stdcall IsClockwise(int dwBaseA,int dwIndexB,int dwIndexC)

Figure 9: C++ declaration of IsClockwise

.text:00401577 push adwIndices
.text:0040157D push adwIndices+4
.text:00401583 mov ecx, 2
.text:00401588 mov dwNumValidIndices, 2
.text:00401592 jmp short loc_4015C6
.text:00401594 ; ---------------------------------------------------------------
.text:00401594
.text:00401594 loc_401594: ; CODE XREF: Update+CB
.text:00401594 push ecx
.text:00401595 mov ecx, adwIndices[ecx*4]
.text:0040159C push ecx ; dwIndexC
.text:0040159D push dword ptr [esp+8] ; dwIndexB
.text:004015A1 push dword ptr [esp+10h] ; dwBaseA
.text:004015A5 call IsClockwise
.text:004015AA pop ecx
.text:004015AB or eax, eax
.text:004015AD jz short loc_4015BF
.text:004015AF push adwIndices[ecx*4]
.text:004015B6 inc ecx
.text:004015B7 inc dwNumValidIndices
.text:004015BD jmp short loc_4015C6
.text:00401594 ; ---------------------------------------------------------------
.text:004015BF
.text:004015BF loc_4015BF: ; CODE XREF: Update+AC
.text:004015BF pop edx
.text:004015C0 dec dwNumValidIndices
.text:004015C6
.text:004015C6 loc_4015C6: ; CODE XREF: Update+91d
.text:004015C6 ; Update+BC
.text:004015C6 cmp ecx, dwNumVertices
.text:004015CC jb short loc_401594
.text:004015CE mov ecx, dwNumValidIndices
.text:004015D4
.text:004015D4 loc_4015D4: ; CODE XREF: Update+DC
.text:004015D4 dec ecx
.text:004015D5 pop edx
.text:004015D6 mov adwValidIndices[ecx*4], edx
.text:004015DD jnz short loc_4015D4

Figure 10: Check: .text:00401577-.text:004015DF

;(dwNumVertices*(dwNumVertices-1)+2)&7==dwNumValidIndices
.text:0040145A mov edx, dwNumVertices
.text:00401460 imul edx, edx
.text:00401463 sub edx, dwNumVertices
.text:00401469 add edx, 2
.text:0040146C and edx, 7
.text:0040146F mov ecx, dwNumValidIndices
.text:00401475 cmp edx, ecx
.text:00401477 jnz BadBoy

Figure 11: The first requirement for a valid polygon:

.text:0040145A-texttt.text:0040147D
.text:0040146F mov ecx, dwNumValidIndices
.text:00401475 cmp edx, ecx
.text:00401477 jnz BadBoy
.text:0040147D fldz
.text:0040147F dec ecx
.text:00401480
.text:00401480 loc_401480: ; CODE XREF: Check+5C
.text:00401480 mov esi, dwNumValidIndices[ecx*4]
.text:004016F3 ; adwValidIndices[ecx*4-4]
.text:00401487 mov edi, adwValidIndices[ecx*4]
.text:0040148E fild aVertices.x[esi*8]
.text:00401495 fisub aVertices.x[edi*8]
.text:0040149C fild aVertices.y[esi*8]
.text:004014A3 fiadd aVertices.y[edi*8]
.text:004014AA fmulp st(1), st
.text:004014AC faddp st(1), st
.text:004014AE dec ecx
.text:004014AF jnz short loc_401480
.text:004014B1 mov ecx, dwNumValidIndices
.text:004014B7 dec ecx
.text:004014B8 mov esi, adwValidIndices[ecx*4]
.text:004014BF mov edi, adwValidIndices
.text:004014C5 fild aVertices.x[esi*8]
.text:004014CC fisub aVertices.x[edi*8]
.text:004014D3 fild aVertices.y[esi*8]
.text:004014DA fiadd aVertices.y[edi*8]
.text:004014E1 fmulp st(1), st
.text:004014E3 faddp st(1), st
.text:004014E5 fidiv dword_4033F6
.text:004014EB fild dwWindowSize_2 ; Width*Height/8
.text:004014F1 fild dwWindowSize ; Width*Height/4
.text:004014F7 fcomip st, st(2)
.text:004014F9 jbe short BadBoy ; dwWindowSize<=st(2)
.text:004014FB fcomip st, st(1)
.text:004014FD jnb short BadBoy ; st(1)>=dwWindowSize_2

Figure 12: The second requirement for a valid polygon:

.text:0040146F-texttt.text:004014FF
float f=0.0
for(int i=0;i<dwNumValidIndices-1;++i)
{
f+=(aPoints[adwValidIndices[i]].x-aPoints[adwValidIndices[i+1]].x)*
(aPoints[adwValidIndices[i]].y+aPoints[adwValidIndices[i+1]].y);
}
f+=(aPoints[adwValidIndices[dwNumValidIndices-1]].x-aPoints[adwValidIndices[0]].x)*
(aPoints[adwValidIndices[dwNumValidIndices-1]].y+aPoints[adwValidIndices[0]].y);
f/=2.0;
if(dwWindowsSize_2<f && //dwWindowSize_2==Width*Height/8
f<dwWindowsSize) //dwWindowSize==Width*Height/4
{
//Good boy
}
else
{
//Bad boy
}

Figure 13: C++ pseudo implementation of the second requirement for a valid polygon
Figure 14: Good boy
 i 0 1 2 3 4
adwIndices[i] 1 2 4 3 5

Table 1: The result of the call to SortAdwIndices with the vertices seen in 8

dwNumVertices dwNumValidIndices
7 4
10 4
12 6
13 6
15 4
18 4
20 6
21 6
23 4
26 4
28 6
29 6
31 4

Table 2: dwNumVertices and dwNumValidIndices combinations which meet the first

requirement

i 0 1 2 3 4 5 6
adwIndices[i] 0 6 5 4 2 3 1

Table 3: adwIndices of the polygon seen in 14

 i 0 1 2 3
adwValidIndices[i] 0 6 2 1

Table 4: adwValidIndices of the polygon seen in 14