Key:

*-An important Item that needs to be finished

B-Initial of person who needs to fix this (Binki)

M-LordofWar needs to fix this

E-normaldotcom needs to fix this

x-finished task

=-Long-term goal

?-needs reconsideration/explanation/rewording

Notes:

  Initial release will no longer feature SSH auth, distrend will handle auth telnet-style. User registration will be done via PHP.

Build

General

=-documenation: manpages

Master

*-Complete testing with interactive test mode

 -update frame assigner to distribute some frames for lower-priority jobs when there's a higher priority job

?-make frame assigner not assign frames beyond the amount of frames in a single job

B-Add calls to remotio -> make API for sending messages ~= a function for every type of message defined in protocol.h

 -Write a stub for getting info from the tarball/validifying the tarball. Read distren-job.xml, a file in the tarball, to find out 1. which rendering system to use (that system, e.g. blender/povray, can read more specifics, such as name of file to pass to blender and frames. Options common between different systems will be handled in common as best as possible)

 -Write a stub for publishing file and constructing job description so that the job can be shared

*-Other XML-writing/reading such as current stats / jobs / data / etc, xml writing for job creation, ...

=-XML-RPC cgi interface for obtaining info

=-charsets: the program should operate in UTF-8. This is partially required by libxml2's (and XML's) usage of UTF-8. We want the program's internal charset to be UTF-8 and figure out if we need w_char/whatever to fulfill UTF-8. I don't believe we do... mabye we just need a declaration that al data stored must be in UTF-8 format. -> we need to consider how charsets work when and if issues come up

M-explain members of struct general_info; e.g., what is timestamp used for, what's the point of the hibernate var, etc.

Slave

 -Fix current bugs

*-Add calls to remoteio, once it works or even before it works -> write message API functions and API

=-Make code more flexible for different types of jobs and operating systems (such as paths, libs, etc)

=-Add code to allow pausing/resuming of the slave (including a running blender process) -> signal handling (SIGTSTP vs. SIGSTOP), distrenslave_control cmdline interface

B-Add tmpdir variable (just like the datadir variable) that is compile-time or whatever -> simple API to allow configuration-file override while taking into account defaults set by ./configure

*-Finish stub to delete old job data and output

=-Upload finished frames' output files while rendering subsequent frames

Options

=-Rewrite some stuff, try to make it simpler

x-Move server and client confuse code into their individual files, rather than in the common file. -- note: there wasn't any client code in options.c. There was the ``server'' section. That section is passed multiple times and provides information to remoteio on how to connect to servers

=-Review all confuse interfacing

B-Push patch for relative includes in confuse to the confuse developers -- windows/mingw32 testing required first

?-Expand execio() to support stderr

/*

 A client is essentially the libdistren library or any app that is submitting 

 files. Thus, it should only need to know which server to access. Credentials

 specific to a server will be specified per-server for now.

*/

client

{

}

include("distrencommon.conf")

/*

  Configuration file for distren.

  Currently, this file is being prepared as the goal for this project. For instance, the ability to support connecting and communicating with servers is suggested by the server sections.

*/

/*

server protofusion

{

  hostname = "protofusion.org"

  username = "distrenc"

  method = "ssh"

  types = {"submit", "distribution"} /* submit means it accepts jobs, distribution means it can host files */

}

server ohnopublishing

{

  hostname = "ohnopublishing.net"

  username = "distrenc"

  method = "ssh"

  types = {"distribution"}

}

/*

 configuration for _this_ server

*/

daemon

{

}

include("distrencommon.conf")

slave

{

  username = "!username"

}

include("distrencommon.conf")

slave

{

  username = "!username"

}

/*

  Copyright 2009 Nathan Phillip Brink, Ethan Zonca, Matthew Orlando

  This file is a part of DistRen.

  DistRen is free software: you can redistribute it and/or modify

  it under the terms of the GNU Affero General Public License as published by

  the Free Software Foundation, either version 3 of the License, or

  (at your option) any later version.

  DistRen is distributed in the hope that it will be useful,

  but WITHOUT ANY WARRANTY; without even the implied warranty of

  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

  GNU Affero General Public License for more details.

  You should have received a copy of the GNU Affero General Public License

  along with DistRen.  If not, see <http://www.gnu.org/licenses/>.

*/

#include "asprintf.h"

#include "protocol.h"

#include "options.h"

#include "remoteio.h"

#include "slavefuncs.h"

#include <stdio.h>

#include <stdlib.h>

#include <string.h>

#include <sys/stat.h>

#include <unistd.h>

int main(int argc, char *argv[])

{

      {

	return 2;

      }

  /* Get conf data */

  char *datadir;

  cfg_t * my_cfg;

  cfg_opt_t myopts[] = {

    CFG_SIMPLE_STR("username", &username),

    CFG_SIMPLE_STR("datadir", &datadir),

    CFG_END()

  };

  struct options_common *commonopts;

  username = NULL;

  options_init(argc, argv, &my_cfg, myopts, "slave", &commonopts);

  /* Notifies the user if there no username in .conf */

  if(username == NULL || strcmp(username, "!username") == 0 )

    fprintf(stderr, "\nPlease ensure that your username is present in distrenslave.conf\n");

  else 

    if( username != NULL || strcmp(username, "!username") != 0 )

      {

	// Log in the user

	if(login_user(username) == 1){

	  fprintf(stderr, "You should now be logged into distren.\n");

	}

	else

	  {

	    fprintf(stderr, "Login failed, please check your username. If you have not registered, please do so on the DistRen website.\n");

	    return 0;

	  }

      }

    else

      {

	fprintf(stderr, "Please check your distrenslave.conf, it appears to be incorrectly formatted.\n");

      }

  int jobnum = 0;

  int framenum = 0;

  char *urltoTar;      /* Full URL to the server-side location of job#.tgz */

  char *pathtoTar;     /* Full path to the location of the job#.tgz */

  char *urltoOutput;   /* Full URL where output is posted */

  char *pathtoJobfile; /* Full path to the job's main file */

  char *pathtoXml;     /* Full path to the job's xml file */

  char *pathtoOutput;  /* Full path to the output (rendered) file */

  char *outputExt;     /* Output Extension (e.g., JPG) */

  char *tarcmd;        /* Command to run for tar. Migrate to libtar sometime */

  char *outdir;        /* Output Directory for tar */

  char *jobdatapath;   /* Path to job data */

  struct distrenjob *myjob; /* Structure to hold data gathered from the XML file */

  int haveWork = 0;

  while(1)

    {

    tell_the_server(0); // Give me some work!

    /* If we got a frame */

    if(haveWork)

      {

        /* @TODO: Add remotio hooks */

        // jobnum = remoteio_read(jobnum); /* Set jobnum from remoteio (we could use info from struct, but we need this info to download the xmlfile */

        //framenum = remoteio_read(jobnum); /* Set framenum from remoteio */

        fprintf(stderr, "Preparing to render frame %d in job %d\n", framenum, jobnum);

        // Variable Preparation

        _distren_asprintf(&jobdatapath, "job%d", jobnum);

        _distren_asprintf(&urltoTar, "http://protofusion.org/distren/stor/job%d/job%d.tar.gz", jobnum); // Prepares URL to download from

        _distren_asprintf(&pathtoTar, "%s/stor/jobdata/job%d.tar.gz", datadir, jobnum); // Prepares destination to save to

        _distren_asprintf(&pathtoJobfile, "%s/%s/job.blend", datadir, jobdatapath ); // Prepares the path to the jobfile

        _distren_asprintf(&urltoOutput, "http://protofusion.org/distren/stor/tmp/", jobdatapath ); // Prepares the URL where output is posted

        _distren_asprintf(&pathtoXml, "%s/job%d/job%d.xml", datadir, jobnum ); // Prepares the path to the job's XML file

        free(jobdatapath);

        // Prepare to download the job tar if it isn't  present

        struct stat buffer;

        int fstatus = stat(pathtoJobfile, &buffer);

        if(fstatus == -1)

          {

            // Download the Tar

            if( curlget(urltoTar, pathtoTar) == 0){

              fprintf(stderr, "Job data retrieved successfully\n");

              free(urltoTar);

            }

            else {

              fprintf(stderr, "Downloading job data from server failed. Check your network connection.\n");

            }

          }

        else

          fprintf(stderr, "Using cached job file...\n");

        mkdir(outdir, 0750);

        system(tarcmd);

        free(tarcmd);

        free(pathtoTar);

        free(outdir);

        /* Parses a job's XML file, puts data in the myjob struct */

        if(xml2distrenjob(&myjob, pathtoXml) == 0)

          {

            fprintf(stderr, "Well, the XML craziness may have worked. Maybe. \n");

            free(pathtoXml);

            /* Frees things up if it was successful. xml2distrenjob() really (usually) only fails if malloc'ing inside it fails */

            distrenjob_free(&myjob);

          }

        else

          {

            fprintf(stderr, "I think the XML craziness may have failed, so I'll terminate just for fun.\n");

            return 1;

          }

        /* Variable-fillers which require XML */

        /* Prepares the path to the jobfile */

        _distren_asprintf(&pathtoOutput, "%s/job%d/output/job%d-frame%d.%s", datadir, jobnum, jobnum, framenum, outputExt );

        free(outputExt);

        /* Execute blender */

            fprintf(stderr,"Error running Blender. Check your installation and/or your PATH.");

            return 1;

        }

        free(pathtoJobfile);

        fprintf(stderr, "Finished frame %d in job %d, uploading...", framenum, jobnum);

          fprintf(stderr,"Upload successful, removing old output...\n");

          remove(pathtoOutput); // Delete the file after its uploaded

        }

          fprintf(stderr,"Upload failed. Check your network connection. Trying again...");

          int tries=0;

              fprintf(stderr, "Upload failed. Trying again in 10 seconds... (attempt %d of 10)", tries);

              tries++;

              sleep(10);

          }

          // @TODO: Keep track of files that we were unable to upload, and upload them later

        }

        free(urltoOutput);

        free(pathtoOutput);

        tell_the_server(DISTREN_REQUEST_DONEFRAME); /* Tells the server that it's done rendering, and upload is done */

      }

    else

      fprintf(stderr,"Nothing to do. Idling...\n");

    // If the server says that every frame for the last jobnum is finished, OR if the data is getting old

      // Note: individual frames are already deleted after uploading,

      // except for ones that couldn't be uploaded

      delete_jobdata(jobnum, datadir);

    }

    sleep(5); // Poll 5 seconds. @TODO: Remove all polling

  }

  return 0;

}

/*

  Copyright 2009 Nathan Phillip Brink, Ethan Zonca, Matthew Orlando

  This file is a part of DistRen.

  DistRen is free software: you can redistribute it and/or modify

  it under the terms of the GNU Affero General Public License as published by

  the Free Software Foundation, either version 3 of the License, or

  (at your option) any later version.

  DistRen is distributed in the hope that it will be useful,

  but WITHOUT ANY WARRANTY; without even the implied warranty of

  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

  GNU Affero General Public License for more details.

  You should have received a copy of the GNU Affero General Public License

  along with DistRen.  If not, see <http://www.gnu.org/licenses/>.

*/

#include "asprintf.h"

#include "distrenjob.h"

#include "execio.h"

#include "protocol.h"

#include "slavefuncs.h"

#include <curl/curl.h>

#include <curl/easy.h>

#include <curl/types.h>

#include <stdio.h>

#include <string.h>

#include <unistd.h>

#include <stdlib.h>

#include <sys/stat.h>

#include <fcntl.h>

/**

 utility function for XPath-ish stuff:

 */

xmlNodePtr xml_quickxpath(xmlXPathContextPtr xpathctxt, xmlChar *path)

{

  xmlNodePtr toreturn;

  xmlXPathObjectPtr xmlxpathobjptr;

  xmlxpathobjptr = xmlXPathEval(path, xpathctxt);

  if(!xmlxpathobjptr

     || !xmlxpathobjptr->nodesetval->nodeNr)

    {

      fprintf(stderr, "XPath resolution failed for ``%s'' in ``%s'' (``%s'')\n", path, xpathctxt->doc->name, xpathctxt->doc->URL);

      return (xmlNodePtr)NULL;

    }

  toreturn = *(xmlxpathobjptr->nodesetval->nodeTab);

  xmlXPathFreeObject(xmlxpathobjptr);

  return toreturn;

}

/** Stub for deleting job data from the disk. @TODO: unstubify me! */

int delete_jobdata(int jobnum, char *datadir)

{

  char *jobpath;

  _distren_asprintf(&jobpath, "%s/%d", datadir, jobnum);

  // rmdir(jobpath);

  fprintf(stderr, "Please manually remove %s. Automatic removal is currently not implemented.\n", jobpath);

  return 0;

}

/** Stub @TODO: Replace with remotio */

void tell_the_server(int stuff){

}

/** Function referenced by curlget() to write data to disk. */

size_t curl_writetodisk(void *ptr, size_t size, size_t nmemb, FILE *stream)

 {

    return fwrite(ptr, size, nmemb, stream);

  }

/** Helper function for cURL's progress display */

int curl_progress( char *Bar,double t,double d,double ultotal,double ulnow)

{

fprintf(stderr,"Downloading: %f%% complete\r",d/t*100);

return 0;

}

/** Retrieves a URL with cURL and saves it to disk */

int curlget(char *url, char *out){

  fprintf(stderr,"Preparing to download %s",url);

  double *Bar; // Stores cURL progress display info

  CURL *curl;

  CURLcode res;

  FILE *outfile;

  curl = curl_easy_init();

  if(curl) {

	outfile = fopen(out, "w"); // Open where we're writing to

  curl_easy_setopt(curl, CURLOPT_URL, url);

  curl_easy_setopt(curl, CURLOPT_WRITEDATA, outfile);

  curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_writetodisk); // this MUST be set for win32 compat.

  curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0);

  curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, curl_progress);

  curl_easy_setopt(curl, CURLOPT_PROGRESSDATA, &Bar);

  res = curl_easy_perform(curl);

  curl_easy_cleanup(curl);

  }

  fprintf(stderr,"\n"); // Clears out he progressbar's carriage return

  return res; // 0 is OK, 1 is 404 or other error

}

/** Posts a file to a url with cUrl */

int curlpost(char *filename, char *url){

  char *targetname = "uploadedfile"; // Name of the target in the php file on the server (Don't change me unless you have different PHP code)

  CURL *curl;

  CURLcode res;

  struct curl_httppost *formpost=NULL;

  struct curl_httppost *lastptr=NULL;

  struct curl_slist *headerlist=NULL;

  static const char buf[] = "Expect:";

  curl_global_init(CURL_GLOBAL_ALL);

  /* upload field... */

  curl_formadd(&formpost,

               &lastptr,

               CURLFORM_COPYNAME, targetname,

               CURLFORM_FILE, filename,

               CURLFORM_END);

  /* filename field... */

  curl_formadd(&formpost,

               &lastptr,

               CURLFORM_COPYNAME, "filename",

               CURLFORM_COPYCONTENTS, filename,

               CURLFORM_END);

  /* submit field, not usually needed, just in case... */

  curl_formadd(&formpost,

               &lastptr,

               CURLFORM_COPYNAME, "submit",

               CURLFORM_COPYCONTENTS, "send",

               CURLFORM_END);

  curl = curl_easy_init();

  headerlist = curl_slist_append(headerlist, buf);

  if(curl) {

    /* Setting the URL to get the post, and the contents of the post */

    curl_easy_setopt(curl, CURLOPT_URL, url);

    curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);

    res = curl_easy_perform(curl);

    curl_easy_cleanup(curl);

    /* cleanup the formpost junk */

    curl_formfree(formpost);

    curl_slist_free_all (headerlist);

  }

  return res;

}

/** Logs the user into the server after ensuring that keys exist */

int login_user(char *username)

{

  // @TODO: Put some telnet-style auth code here

  return 1; // success

}

  int maxlinelen = 120;

  char buffer[maxlinelen+2];

  char *buff_ptr, *find_ptr;

  FILE *fp1, *fp2;

  fp1 = fopen(fileOrig,"r");

  fp2 = fopen(fileRepl,"w");

  if (fp1 ==NULL){

    fprintf(stderr, "%s doesn't exist\n",fileOrig);

    return 0;

  }

  else if(fp2 ==NULL){

    fprintf(stderr, "Can't write a file to disk! Check permissions.\n");

    return 0;

  }

  else{

    while(fgets(buffer,maxlinelen+2,fp1))

      {

        buff_ptr = buffer;

        {

           while(buff_ptr < find_ptr)

             fputc((int)*buff_ptr++,fp2);

           buff_ptr += find_len;

         }

         fputs(buff_ptr,fp2);

      }

    rename(fileRepl, fileOrig);

  }

  fclose(fp2);

  fclose(fp1);

  fprintf(stderr,"Wrote conf file...\n");

return 1; // Success

}

/* Executors */

/** Executor function for Blender operations */

int exec_blender(char *input, char *output, int frame)

{

  fprintf(stderr,"Preparing to execute...\n");

  int ret;

  char *frame_str;

  char buf[10];

  struct execio *testrem;

  size_t readlen;

  _distren_asprintf(&frame_str, "%i", frame);

  ret = execio_open(&testrem, command, cmd);

  buf[9] = '\0';

  while(!execio_read(testrem, buf, 9, &readlen))

    {

      if(readlen > 9) {

        fprintf(stderr, "Something is terribly wrong!\n");

       }

       buf[readlen] = '\0';

       fprintf(stderr, "read \"%s\"\n", buf);

    }

  execio_close(testrem);

  return ret;

}

void xmlinit()

{

  xmlInitParser();

  xmlXPathInit();

}

void xmlcleanup()

{

  xmlCleanupParser();

}

/*

  Copyright 2009 Nathan Phillip Brink, Ethan Zonca, Matthew Orlando

  This file is a part of DistRen.

  DistRen is free software: you can redistribute it and/or modify

  it under the terms of the GNU Affero General Public License as published by

  the Free Software Foundation, either version 3 of the License, or

  (at your option) any later version.

  DistRen is distributed in the hope that it will be useful,

  but WITHOUT ANY WARRANTY; without even the implied warranty of

  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

  GNU Affero General Public License for more details.

  You should have received a copy of the GNU Affero General Public License

  along with DistRen.  If not, see <http://www.gnu.org/licenses/>.

*/

#ifndef _DISTREN_SLAVEFUNCS_H

#define _DISTREN_SLAVEFUNCS_H

#include "distrenjob.h"

#include <libxml/xpath.h>

#include <stdio.h>

xmlNodePtr xml_quickxpath(xmlXPathContextPtr xpathctxt, xmlChar *path);

int software_updatecheck();

void tell_the_server(int stuff);

int delete_jobdata(int jobnum, char *datadir);

size_t curl_writetodisk(void *ptr, size_t size, size_t nmemb, FILE *stream);

int curlget(char *url, char *out);

int curlpost(char *filename, char *url);

int ssh_keygen();

int register_user(char *username, char *email);

int login_user(char *username);

int exec_blender(char *input, char *output, int frame);

void xmlinit();

void xmlcleanup();

#endif