Calling Pexec with command line bigger than 124 chars

[JeanMars]

Hi,
since atari.org moved to https, I realized that Vision's updater from Internet is broken as:
- curl is to be used (Mint's wget does not feature TLS)
- There are more parameters to pass to curl and so the command line is usually longer than 124 characters which fails this call:
Pexec( 0, full_name, pascal_cmdline, NULL ) as pascal_cmdline is bigger than 124 bytes

I read about ARGV procedure at https://exxosforum.co.uk/atari/mirror/t ... 05011.html but it looks unclear to me; is there any code sample that already uses this so I can get inspired from ?

Thanks,
Jean

[czietz]

The original / official ARGV specification contains some sample code:
https://docs.dev-docs.org/htm/search.php?find=argv

PS: Even though it's not relevant for this discussion, the current (i.e., still maintained) version of tos.hyp is at: https://freemint.github.io/tos.hyp/en/index.html

[JeanMars]

Thanks,
Exactly what I was looking for.
Cheers,
Jean

[ThorstenOtto]

You may also look at the implementation of spawnve from mintlib: https://github.com/freemint/mintlib/blo ... /spawnve.c

Edit: another solution would be to execute a script instead of executing curl directly, by doing something like Pexec(0, "/bin/sh", "-c <your-script-name>", NULL);

[JeanMars]

Hi Thorsten,

still a bit confused by this ARGV parameters to pass. Basically I want to execute the following command line:
u:\usr\bin\curl -L --cacert cacert.pem --get https://vision2.atari.org/vupdate/staging/vision.ver --output H:\PURE_C\PROJECTS\VISION\UPDATE\vision.ver
As it is longer than 124 chars, I'm calling XPexec like this:
Pexec( 0, childname, tail, childenv ) ;
Where childname=u:\usr\bin\curl\0 (zero terminated string)
tail=-L --cacert cacert.pem --get https://vision2.atari.org/vupdate/staging/vision.ver --output H:\PURE_C\PROJECTS\VISION\UPDATE\vi\0 (truncated to 124 chars)
childenv is a set of strings separated by \0 and terminated by \0\0(ARGV=\0u:\usr\bin\curl\0-L\0--cacert\0...vision.ver\0\0):
arg0:ARGV=
arg1:u:\usr\bin\curl
arg2:-L
arg3:--cacert
arg4:cacert.pem
arg5:--get
arg6:https://vision2.atari.org/vupdate/staging/vision.ver
arg7:--output
arg8:H:\PURE_C\PROJECTS\VISION\UPDATE\vision.ver

But curl returns:
curl (27): Out of memory

I'm surely doing something wrong, if anyone can pinpoint the error, that will be nice
Cheers,
Jean

PS: I can also go with your suggestion (script), but I would prefer avoid creating a temporary file somewhere.

[ThorstenOtto]

You can try the attached version of the spawn function, ported to Pure-C. Some notes:
- since Pure-C does not provide an "environ" array, the signature of the "spawnve" function is not posix-like, and i renamed it to pc_spawnve, to avoid potential clashes with other include files
- the code that looks at the "PCONVERT" environment variable has been removed, since the startup code of Pure-C does not do the dos->unix conversion that was undone there
- Calls to _unx2dos have also been removed, so you must pass dos style pathnames and arguments (\\, not /)
- you can disable the code that looks for shell scripts by undefining HASH_BANG, if you don't need it

For use in your project, you should copy the prototypes and the defines at the top of the file to some of your include files (in mintlib, they are in <process.h>)

[czietz]

Do you pass 127 as the length byte of the command line to the Pexec call?

[joska]

Edit: another solution would be to execute a script instead of executing curl directly, by doing something like Pexec(0, "/bin/sh", "-c <your-script-name>", NULL);

I don't have much experience with curl, but I believe it's possible to put the POST/GET parameters in a file and pass the filename to curl instead of the actual parameters. That should shorten the command line a bit.

[JeanMars]

Hi all,

Do you pass 127 as the length byte of the command line to the Pexec call?

Yes, actually my code is very similar to the reference one at https://docs.dev-docs.org/htm/search.php?find=argv

Code: Select all

  *ptail  = 0 ;
  tail[0] = 127 ;

BTW I'm puzzled that is set t 127 even if size i smaller, I would have set it to the actual size of the command instead if <= 124

Thanks Thorsten for the sample code, I'll check how I can use it for Vision.

But right now I need some time to recover from my flu

Happy new year to everyone!
Cheers,
Jean

[JeanMars]

At least I have the explanation for this:

BTW I'm puzzled that is set t 127 even if size i smaller, I would have set it to the actual size of the command instead if <= 124

From https://docs.dev-docs.org/htm/search.php?find=argv:

As an aside, the Pexec Cookbook erroneously implies that a command line can be 126 or 127 characters long. In fact, GEMDOS only copies to the
child's basepage up to 125 bytes, or until it encounters a null, from the argument string passed to Pexec(). It ignores the length byte, placing a null at the same place it found one or at the 126th byte if no null is found.

[JeanMars]

Hi again,

I don't have much experience with curl, but I believe it's possible to put the POST/GET parameters in a file and pass the filename to curl instead of the actual parameters. That should shorten the command line a bit.

Not so sure and anyway if I have to use file, better put the whole command line directly in s a script as Thorsten suggested.

I don't have cstartv.o from PURE C; can someone please give a link to it?

Thing is that I don't see what the callee is expecting in env variable of Pexec.
For example, the command line
u:\usr\bin\curl -L --cacert cacert.pem --get <url> --output <pathname>
Is env supposed to be an array of pointers to:
argv[0]=ARGV=
argv[1]=u:\usr\bin\curl
argv[2]=-L
argv[3]=--cacert
argv[4]=-cacert.pem
argv[5]=--get
argv[6]=<url>
argv[7]=--output
argv[8]=<pathname>

I've read some source code like toswin2, spawn, .. but still unclear and don' understand why my code is not working. I have attached it. Unfortunatly, I need Aranym to run this where debug capabilities are limited to log file...

Cheers,
Jean

[joska]

https://stackoverflow.com/questions/549 ... -in-a-file

If you use a shell-script you will also need a shell, which you don't if you pass the parameters in a file. Passing the parameters in a file will also hide them from the commandline (which is visible with ps/top or even the XaAES task manager).

[JeanMars]

Not sure I can use this method; it seems to be interpreted by the server not handled by the local curl process:
I need to call curl like this:
u:\usr\bin\curl -L --cacert cacert.pem --get https://vision2.atari.org/vupdate/staging/vision.ver --output H:\PURE_C\PROJECTS\VISION\UPDATE\vision.ver
There is no data to pass to the URL itself, just instruct curl to use a provided RootCA certificate and to store the outcome into a local file.
If I understand correctly, using -G and -d pass data as variables to the server (https://curl.se/docs/manpage.html#-G):

(HTTP) When used, this option makes all data specified with -d, --data, --data-binary or --data-urlencode to be used in an HTTP GET request instead of the POST request that otherwise would be used. curl appends the provided data to the URL as a query string.

And I'm still annoyed by not being able to make this quite simple piece of code work without going through a shell and/or additional file.

What a mess for calling Pexec with long parameters...

[joska]

You can cwd to the destination directory before calling curl. Dropping the directory part of the "--output" parameter will get your commandline under 126 characters. You can also use the short versions of the parameters (like -o instead of --output) to further shorten the commandline.

[JeanMars]

Hi Joska,

of course with a little bit of tricking here, I can save some bytes and end-up with less than 124 bytes for command line. But that would be not a so nice way to solve the issue (and fix the annoying bug).

[ThorstenOtto]

BTW I'm puzzled that is set t 127 even if size i smaller, I would have set it to the actual size of the command instead if <= 124

Yes, that is done because if you only pass the parameter in the commandline, argv[0] in the application will be empty (or some arbitrary string), and some programs need this, because they behave differently depending under which name they are invoked.

Unfortunately there is no easy way to know whether that is needed or not. Not always using ARGV will break such applications. OTOH, always using it will break applications that have a startup code that does not understand ARGV. In mintlib it is done on the assumption that you are almost always spawning programs which are itself unix ports and compiled with mintlib. That discrepancy is also the reason why most desktops have an option whether to use ARGV or not (but of course in your case, it is out of control of the desktop).

I don't have cstartv.o from PURE C; can someone please give a link to it?

Oops, sorry, But it is only referenced in the sample. You can use the one from magicmac, or simply replace it by pcvstart.o from Pure-C.

[joska]

- since Pure-C does not provide an "environ" array,

Not quite sure what you mean by "environ array", but Pure C's pcvstart.o pass three parameters to main - argc, **argv and **envp. So..

Code: Select all

int main(int argc, char **argv, char **envp)
{
   int i = 0;

   while (envp[i])
      printf("%s\n", envp[i++]);

   return 0;
}

...will print a list of all environment variables.

[medmed]

Hi,

In the mean time Why you don't replace your long url by $(cat filewithurlandargsinside.txt) with a filename containing one char? Like curl $(cat u)...

Another solution would be to build wget with TLS support.

Or also build a custom exe with libcurl with everything inside the exe.
I had started to write some code related to it here.

[medmed]

Duplicate

[ThorstenOtto]

Not quite sure what you mean by "environ array", but Pure C's pcvstart.o pass three parameters to main - argc, **argv and **envp. So..

I mean the global "environ" variable, which does not exist, but is required by the original implementation in mintlib. For the same reason, there is also nothing like putenv() or setenv(). And getenv() does not use any global variable, but looks at _BasPag->p_env.

All that could be implemented of course, but then you'll need a custom startup module. Or put the "envp" that is passed to main in that global variable. That should work too, but then you must not forget to do that assignment.

[ThorstenOtto]

In the mean time Why you don't replace your long url by $(cat filewithurlandargsinside.txt) with a filename containing one char? Like curl $(cat u)...

That $() in only expanded by a shell, so you would have to run a shell instead of curl. And of course you also need the cat command.

[JeanMars]

I made the changes to go through a script ran by sh. Did try many options with ARGV but never came to a success. I used the spawn code provided by Thorsten to add environment which made some progress (no more error 27 out of memory) but outfile file is reporting document moved, just like if -L option is not present. However, it is in the argv list.
Giving up, I will let sh deal with this funky way of passing ARGV parameters

[JeanMars]

Just to respond to medmed (thanks BTW!):
using easy curl would be even better as it would avoid instancing a new program...at the cost of a static library added to Vision. Not sure how big this library is but it would increase Vision's executable quite largely; which I'd like to avoid as this curl process would just be called for checking updates/updating. I could go with a dynamic library but too complicated for the intent IMHO. Especially compared to the "simple" problem of just being able to Pexec with more than 124 bytes as cmdline
wget is an external tool to Vision, and it would be dfficult to control from Vision whether or not wget features ssl.
Cheers,
Jean

[medmed]

Just to respond to medmed (thanks BTW!):
using easy curl would be even better as it would avoid instancing a new program...at the cost of a static library added to Vision. Not sure how big this library is but it would increase Vision's executable quite largely; which I'd like to avoid as this curl process would just be called for checking updates/updating. I could go with a dynamic library but too complicated for the intent IMHO. Especially compared to the "simple" problem of just being able to Pexec with more than 124 bytes as cmdline
wget is an external tool to Vision, and it would be dfficult to control from Vision whether or not wget features ssl.
Cheers,
Jean

Hi Jean,

Yes. Thanks for the reply I was thinking to a separate binary who just download the file with all your custom variables inside like timeout, ssl path, url... Vision could then just check for his presence and launch it as is.
It could also read parameters from file. But yes adding it to vision would increase a lot the vision binary size.

[JeanMars]

Hi all,
problem SOLVED thanks to Olivier Landemarre
The root issue is that after "ARGV=", a space followed by \0 is required before passing command line (parameters separated by \0 and changing \ to /).
Here is the routine:

Code: Select all

long Pexec_eas(int mode, char* full_name, char* cmdline)
{
  /* A BIG thank to Olivier Landemarre ! */
  char  buffer[1500] ;
  char  eas_cmd[128] ;
  char* pt_env = buffer ;
  char* source ;
  char* envp = _BasPag->p_env ;

  /* Program Environment is required else it fails (e.g. err 27 with curl) */
  while( *envp )
  {
    while ( *envp ) *pt_env++ = *envp++ ;
    *pt_env++ = 0 ;
    envp++ ;
  }

  *pt_env++='A' ;
  *pt_env++='R' ;
  *pt_env++='G' ;
  *pt_env++='V' ;
  *pt_env++='=' ;
  *pt_env++ = 0 ;
  *pt_env++ = ' ' ; /* These 2 are important else */
  *pt_env++ = 0 ;   /* First parameter is ignored ! */
  while ( *cmdline )
  {
    if ( *cmdline == ' ' )       *pt_env++ = 0 ;        /* New parameter/value */
    else if ( *cmdline == '\\' ) *pt_env++ = '/' ;      /* Change \ to / */
    else                         *pt_env++ = *cmdline ; /* Next char of parameter/value */
    cmdline++ ;
  }

  *pt_env++  = 0 ;
  *pt_env    = 0 ;
  eas_cmd[0] = 127 ;
  return Pexec( mode, full_name, eas_cmd, buffer ) ;
}

No overflow control yet, just for validating it works...
@medmed: so no more need for an external program either

Cheers
Jean