Open the archive in a hex editor and try to look for clues in the headers.
Edit: For example UPX packed programs have this near the beginning:
0160h: 4C DF 7F FE 20 08 67 02 9F CF 4E F9 0A 00 24 49 Lßþ .g.ŸÏNù..$I
0170h: 6E 66 6F 3A 20 54 68 69 73 20 66 69 6C 65 20 69 nfo: This file i
0180h: 73 20 70 61 63 6B 65 64 20 77 69 74 68 20 74 68 s packed with th
0190h: 65 20 55 50 58 20 65 78 65 63 75 74 61 62 6C 65 e UPX executable
01A0h: 20 70 61 63 6B 65 72 20 68 74 74 70 3A 2F 2F 75 packer
http://u 01B0h: 70 78 2E 73 66 2E 6E 65 74 20 24 0A 00 24 49 64 px.sf.net $..$Id
01C0h: 3A 20 55 50 58 20 33 2E 39 34 20 43 6F 70 79 72 : UPX 3.94 Copyr
01D0h: 69 67 68 74 20 28 43 29 20 31 39 39 36 2D 32 30 ight (C) 1996-20
01E0h: 31 37 20 74 68 65 20 55 50 58 20 54 65 61 6D 2E 17 the UPX Team.
01F0h: 20 41 6C 6C 20 52 69 67 68 74 73 20 52 65 73 65 All Rights Rese
0200h: 72 76 65 64 2E 20 24 0A 00 00 00 00 55 50 58 21 rved. $.....UPX!