DistRen Changeset - e8abe91d635f · protofusion repositories

Changeset - e8abe91d635f

Parent rev.

Child rev.

[Not reviewed]

default

0 3 0

Ethan Zonca (ethanzonca) - 15 years ago 2010-06-26 01:30:23
e@ethanzonca.com

Backported tar to standard slave

3 files changed with 5 insertions and 5 deletions:

TODO

src/server/simpleslave.c

src/server/slave.c

0 comments (0 inline, 0 general)

TODO

➞

Show inline comments

 Key:
 *-An important Item that needs to be finished
 B-Initial of person who needs to fix this (Binki)
 M-LordofWar needs to fix this
 E-normaldotcom needs to fix this
 x-finished task
 =-Long-term goal
 ~-Cancelled goal
 a-Goal needing assessment
 ?-needs reconsideration/explanation/rewording
 Notes:
   Initial release will no longer feature SSH auth, distrend will handle auth telnet-style. User registration will be done via PHP.
   Simpleslave uses a PHP API to communicate with the mysql-backed server, allowing debugging while server software is nonfunctional.
 Build
 General
   =-documenation: manpages
   =-runtime support: should we eventually just go with APR, glib, maybe something from suckless people? Is libdaemon (or whatever's out there) useful?
   =-XML (XML-RPC): useful as a communication protocol? as RPC protocol? see atheme; useful as job description format possibly.
   B-remoteio: figure out how to handle communications; async + callbacks seems the best method to me atm
 Master
   B/E/?-Add calls to remotio -> make API for sending messages ~= a function for every type of message defined in protocol.h (are SendSignal() and SendExtSignal() good enough?)
   a-XML-RPC interface for obtaining info
   =-charsets: the program should operate in UTF-8. This is partially required by libxml2's (and XML's) usage of UTF-8. We want the program's internal charset to be UTF-8 and figure out if we need w_char/whatever to fulfill UTF-8. I don't believe we do... mabye we just need a declaration that al data stored must be in UTF-8 format. -> we need to consider how charsets work when and if issues come up
   *-figure out delete-during-traversal with liblist
   *-after job is complete and verified by the user, tar frames for easy download
   x-Complete testing with interactive test mode
   x-Complete DB testing with php server workaround
   x-resolve conflict between liblist and libmysqlclient
   x-update frame assigner to distribute some frames for lower-priority jobs when there's a higher priority job (Ask LordOfWar if this is implemented in sprocs already(it is))
 Slave
   B-UnpackJob() and supporting functions need to work
   E-Restore tar functionality after unpackJob() is functional
   E-Add calls to remoteio, once it works or even before it works -> write message API functions and API
   =-Make code more flexible for different types of jobs and operating systems (such as paths, libs, etc)
   =-Add code to allow pausing/resuming of the slave (including a running blender process) -> signal handling (SIGTSTP vs. SIGSTOP), distrenslave_control cmdline interface
   B-Add tmpdir variable (just like the datadir variable) that is compile-time or whatever -> simple API to allow configuration-file override while taking into account defaults set by ./configure
   E-Add error checking and cleaner recursive deletion to delete_jobdata
   =-Upload finished frames' output files while rendering subsequent frames
   x=Add lockout of slaves below a certain version
   x-Add simpleslave with code to work around nonexisting server codebase
   x-Backport SimpleSlave functionality to slave code
   x-Restore tar functionality after unpackJob() is functional (6/26/10: unpackJob isn't functional, but tar re-implemented)
 Options
   B-Push patch for relative includes in confuse to the confuse developers -- windows/mingw32 testing required first
   ?-Expand execio() to support stderr
   =-Rewrite some stuff, try to make it simpler
   =-Review all confuse interfacing
   x-Move server and client confuse code into their individual files, rather than in the common file. -- note: there wasn't any client code in options.c. There was the ``server'' section. That section is passed multiple times and provides information to remoteio on how to connect to servers
 WebInterface
   *-Tarball validation and other checks
   *-Scale images on web interface, imagemagick?
   *-Add "stop" button functionality and other missing button functionality
   E-Check queries and output for all edit scripts, e.g., edit profile userlevel dropdown doesn't show the user's current userlevel
   E-Simplify web interface, make it more lightweight
   E-Make some icons for deleting/editing/etc that aren't horribly ugly
   E-Get rid of "My Output" or move some functionality to this area
   =-Allow "archiving" of jobs?
   =-Begin working on MeshLib seamless integration with DistRen
   =-Create install.php to guide through database connection and schema deployment
   =-Add feature to "purchase" priority with Render_Points
 Overall
   *-Output type is currently a placeholder option at this point, make it actually affect the output
   *-Change the way the database handles storage of paths to frame data (path + extension?)
   *-Output resolution is currently a placeholder option, make it actually affect the output

src/server/simpleslave.c

➞

Show inline comments

@@ @@ -173,97 +173,97 @@ int main(int argc, char *argv[]) @@
 	  fprintf(stderr,"Benchmark failed! Exiting.\n");
 	  return 1;
+	}
       else
+	{
 	  fprintf(stderr,"Benchmark successful, time taken was %d seconds, giving you a render power of %d.\n",
 		  benchmarkTime, renderPower);
 	  _web_setrenderpower(slavekey, password, renderPower);
 	  return 0;
+	}
+    }
   if(!DEBUG)
     fprintf(stderr, "Running..");
   // Main loop
   while(!quit)
+    {
     // request work
     if(DEBUG)
       fprintf(stderr,"Requesting work from %s...\n", hostname);
     haveWork = _web_getwork(slavekey, password, &jobnum, &framenum);
     /* If we got a frame */
     if(haveWork)
+      {
         fprintf(stderr,"Got work from server...\n");
         if(DEBUG)
           fprintf(stderr, "Preparing to render frame %d in job %d\n", framenum, jobnum);
         prepareJobPaths(jobnum, framenum, outputExt, datadir, &urltoTar, &pathtoTar, &pathtoTardir, &pathtoJob, &pathtoJobfile, &urltoJobfile, &urltoOutput, &pathtoOutput, &pathtoRenderOutput, &pathtoOutdir);
         //free(outputExt);
         mkdir(pathtoTardir, 0700);
         int dlret = downloadTar(urltoTar, pathtoTar);
         //int dlret = downloadTar(urltoJobfile, pathtoJobfile);
         if(dlret == 0)
           fprintf(stderr,"Data retrieved successfully!\n");
         else if(dlret == 3){
           _web_resetframe(slavekey, password, jobnum, framenum);  // Unassign the frame on the server so other slaves can render it
           return 0; // ouput dir doesn't exist
+        }
         else
           if(DEBUG)
             //fprintf(stderr,"Using existing uncompressed data %s...\n", pathtoJobfile);
             fprintf(stderr,"Using existing tarball %s...\n", pathtoTar);
         // Check if tar exists already
         struct stat jbuffer;
-        int jstatus = stat(pathtoJobfile, &jbuffer);
+        int jstatus = stat(pathtoTar, &jbuffer);
         if(jstatus == -1){
           fprintf(stderr,"Main job file does not exist, extracting...\n");
           // If error unpacking tarball
           if(unpackJob(pathtoJob, pathtoTar)){
             _web_resetframe(slavekey, password, jobnum, framenum);  // Unassign the frame on the server so other slaves can render it
             fprintf(stderr,"Extraction failed, exiting.\n");
             return 1;
+          }
+        }
         /* ignore return because directory may exist already */
         if(DEBUG)
           fprintf(stderr,"Creating output directory %s\n", pathtoOutdir);
         mkdir(pathtoOutdir, 0700);
         if(DEBUG)
           fprintf(stderr,"Marking frame started on server... ");
         _web_startframe(slavekey, password, jobnum, framenum);
         /* Execute blender */
         if(DEBUG){
           fprintf(stderr,"Executing blender on file %s\n", pathtoJobfile);
           fprintf(stderr,"Directing output to file %s\n", pathtoOutput);
+        }
         if(exec_blender(pathtoJobfile, pathtoRenderOutput, framenum))
+          {
             fprintf(stderr,"Error running Blender. Check your installation and/or your PATH.\n");
             _web_resetframe(slavekey, password, jobnum, framenum);  // Unassign the frame on the server so other slaves can render it
             return 1;
+          }
         free(pathtoJobfile);
         pathtoJobfile = NULL;
         struct stat buffer;
         int fstatus = stat(pathtoOutput, &buffer);
         if(fstatus == -1){
           fprintf(stderr,"*** %s doesn't exist! Scene may not have camera, or your blender installation is not working.\n", pathtoOutput);
           _web_resetframe(slavekey, password, jobnum, framenum);  // Unassign the frame on the server so other slaves can render it
           return 1;
+        }
         else{
           /* Post-execution */
           if(DEBUG)
             fprintf(stderr, "Finished frame %d in job %d, uploading...\n", framenum, jobnum);
           else
             fprintf(stderr,"Finished.\n");

src/server/slave.c

➞

Show inline comments

@@ @@ -118,169 +118,168 @@ int main(int argc, char *argv[]) @@
+    }
   if(!username)
+    {
       fprintf(stderr, "username not set\n");
       return 1;
+    }
   if(!password)
+    {
       fprintf(stderr, "password not set\n");
       return 1;
+    }
   if(!hostname)
+    {
       fprintf(stderr, "hostname not set\n");
       return 1;
+    }
   /* Notifies the user if there no username in .conf */
   if(checkUsername(username))
     return 1;
   if(!strncmp(password, "!password",10))
+    {
       fprintf(stderr, "You haven't specified a password. Please edit distrenslave.conf!\n");
       return 1;
+    }
   fprintf(stderr, "Connecting to server...\n");
   if(remoteio_open(&comm_slave, commonopts->remoteio, server))
+    {
       fprintf(stderr, "Error connecting to server; exiting\n");
       return 1;
+    }
   // Variables needed for main loop
   int jobnum = 0;
   int framenum = 0;
   int slavekey = atoi(username); // @TODO: Make this more friendly
   char *urltoTar;      /* Full URL to the server-side location of job#.tgz */
   char *pathtoTar;     /* Full path to the location of the job#.tgz */
   char *pathtoTardir;
   char *urltoOutput;   /* Full URL where output is posted */
   char *pathtoOutput;  /* Full path to the output (rendered) file */
   char *pathtoOutdir;  /* Full path to output directory */
   char *pathtoRenderOutput;  /* Contains blender framenum placeholder */
   // Temporary for uncomressed testing @TODO: remove
   char *urltoJobfile;
   char *urltoJobfile; /* No longer used, url to .blend on server */
   char *pathtoJob;     /* Full path to job data folder */
   char *pathtoJobfile; /* Full path to the job's main file */
   char *outputExt = "jpg";     /* Output Extension (e.g., JPG) */
   int haveWork = 0;
   int quit = 0;
   fprintf(stderr,"\nDistRen Slave Pre-Alpha %s\n- Experimental build: Use at your own risk!\n\n", PACKAGE_VERSION);
   int benchmarkTime = 0;
   int renderPower = 0;
   // @TODO: add call to function to force recalc if $render_power == ""
   if(runBenchmark)
+    {
       if(slaveBenchmark(datadir, &benchmarkTime, &renderPower))
+        {
           fprintf(stderr,"Benchmark failed! Exiting.\n");
           return 1;
+        }
       else
+        {
           fprintf(stderr,"Benchmark successful, time taken was %d seconds, giving you a render power of %d.\n",
                   benchmarkTime, renderPower);
           _web_setrenderpower(slavekey, password, renderPower);
           return 0;
+        }
+    }
   if(!DEBUG)
     fprintf(stderr, "Running..");
   // Main loop
   while(!quit)
+    {
     // request work
     fprintf(stderr,"Requesting work...\n");
     haveWork = getwork(comm_slave, &jobnum, &framenum);
     /* If we got a frame */
     if(haveWork)
+      {
         fprintf(stderr,"Got work from server...\n");
         /* @TODO: Add remotio hooks */
         // jobnum = remoteio_read(jobnum); /* Set jobnum from remoteio (we could use info from struct, but we need this info to download the xmlfile */
         // framenum = remoteio_read(jobnum); /* Set framenum from remoteio */
         // outputExt = remotio)read(outputExt); /* Set output extension from remotio */
         if(DEBUG)
           fprintf(stderr, "Preparing to render frame %d in job %d\n", framenum, jobnum);
         prepareJobPaths(jobnum, framenum, outputExt, datadir, &urltoTar, &pathtoTar,&pathtoTardir,&pathtoJob, &pathtoJobfile, &urltoJobfile, &urltoOutput, &pathtoOutput, &pathtoRenderOutput, &pathtoOutdir);
         free(outputExt);
         int dlret = downloadTar(urltoTar, pathtoTar);
         if(dlret == 0)
           fprintf(stderr,"Data retrieved successfully!\n");
         else if(dlret == 3){
           resetframe(comm_slave, jobnum, framenum);  // Unassign the frame on the server so other slaves can render it
           return 0; // ouput dir doesn't exist
+        }
         else
           if(DEBUG)
             fprintf(stderr,"Using existing tarball %s...\n", pathtoTar);
         // Decompress tarball
         struct stat jbuffer;
-        int jstatus = stat(pathtoJobfile, &jbuffer);
+        int jstatus = stat(pathtoTar, &jbuffer);
         if(jstatus == -1){
           if(DEBUG)
             fprintf(stderr,"Main job file does not exist, extracting...\n");
           // If error unpacking tarball
           if(unpackJob(pathtoJob, pathtoTar)){
             resetframe(comm_slave, jobnum, framenum);  // Unassign the frame on the server so other slaves can render it
             fprintf(stderr,"Error decompressing tarball! Exiting.\n");
             return 1;
+          }
+        }
         /* ignore return because directory may exist already */
         if(DEBUG)
           fprintf(stderr,"Creating output directory %s\n", pathtoOutdir);
         mkdir(pathtoOutdir, 0700);
         if(DEBUG)
           fprintf(stderr,"Marking frame started on server... ");
         _web_startframe(slavekey, password, jobnum, framenum);
         /* Execute blender */
         if(DEBUG){
           fprintf(stderr,"Executing blender on file %s\n", pathtoJobfile);
           fprintf(stderr,"Directing output to file %s\n", pathtoOutput);
+        }
         /* Execute blender */
         if(exec_blender(pathtoJobfile, pathtoOutput, framenum))
+          {
             fprintf(stderr,"Error running Blender. Check your installation and/or your PATH.\n");
             resetframe(comm_slave, jobnum, framenum);  // Unassign the frame on the server so other slaves can render it
             return 1;
+          }
         free(pathtoJobfile);
         pathtoJobfile = NULL;
         struct stat buffer;
         int fstatus = stat(pathtoOutput, &buffer);
         if(fstatus == -1){
           fprintf(stderr,"*** %s doesn't exist! Scene may not have camera, or your blender installation is not working.\n", pathtoOutput);
           resetframe(comm_slave, jobnum, framenum);  // Unassign the frame on the server so other slaves can render it
           return 1;
+        }
         else{
           /* Post-execution */
           if(DEBUG)
             fprintf(stderr, "Finished frame %d in job %d, uploading...\n", framenum, jobnum);

0 comments (0 inline, 0 general)