If you had read the vbdll.txt (or something close to that) you would
know that the function has to use the _stdcall calling convetnion, and
that you need to define it in the DEF file via the exports statement:

#include<math.h>
#define DLLexport _declspec(dllexport)
extern  "C" DLLexport int  _stdcall compHyp(int, int);
int compHyp(int leg1, int leg2)
{
  return (int)sqrt(leg1*leg1+leg2*leg2);

--
Talk is cheap & lies are expensive....
                My wallets fat & so is my head.