Changeset - 12246b166181
[Not reviewed]
default
0 2 0
Nathan Brink (binki) - 15 years ago 2010-08-08 00:19:54
ohnobinki@ohnopublishing.net
Provide a basic model for job distribution.
2 files changed with 12 insertions and 1 deletions:
0 comments (0 inline, 0 general)
doc/architecture.txt
Show inline comments
 
Concepts:
 

	
 
- trust: The first model of distren will assume that is can trust all other servers it associates
 
    with. This removes a great deal of potential complication and yet this should be addressed
 
    in a later revision of this document.
 

	
 
- server: A process that accepts jobs, manages rendering processes, and talks to other servers
 
  - server identification: each server must be given a unique, alphanumeric identifier.
 
      It may make sense to have a server on a machine named after a domain name by which the
 
      server may be accessed. There doesn't need to be such an association, though. This document
 
      assumes that a server's name shall also remain constant (anything else would complicate...).
 
      If a new URL schema would be used, I suppose that distren://<servername>/ might identify a
 
      distren server.
 

	
 
- job: Represents a collection of renderings that must be unbundled, rendered, and bundled together again.
 
  - job identification: A server shall internally assign numeric job identifiers. It must only
 
      make sure that it never issues the same numeric job identifier to multiple jobs. To allow
 
      multiple servers to share jobs with eachother (which is the the whole point of distren),
 
      a job shall be refered to by prefixing the job a server identification. For example,
 
      distren://<servername>/job/<jobid>
 
  - job packaging: Currently, the most creative way of dealing with jobs I have at the moment is
 
      storing all of the data necessary to render the job in a tarball. This tarball can be
 
      treated as a directory of a normal filesystem. A job's directory would contain at least one
 
      file called ``distrenjob.xml'' which provides information necessary to render the job. This
 
      will be an XML file rather than a rigidly defined binary format because XML supports
 
      arbitrary data storage out of the box. This should allow different rendering backends to
 
      store the extra information that is specific to that backend.
 
  - post-render rebundling: Again, to make things simpler, the server where a job is rendered
 
      shall be responsible for collecting the individual completed frames and collecting them
 
      into a tarball. This tarball may be called for using a client. This tarball is a bundle
 
      of completed frames and will exclude the seed tarball.
 
  - job announcement: When a job is created, the server it was submitted to shall make a
 
      global announcement that the job is available. All servers should record this announcement
 
      All servers will maintain a list of globally available jobs. When a server becomes available
 
      to render a frame, it will scan the lists of announcements it has received and then ask
 
      the originating server for a frame.
 

	
 
- frame: Represents a distinct, and hopefully small, unit of work that a renderer backend can perform.
 
  - frame identification: Like a job numeral is useless without a qualifying servername, a 
 
      frame's identification number would be useless without an accompanying job identifier. The
 
      numeric value of a frame identification value must be unique to that job. There are no
 
      restrictions of ordering of frame numbers except that they are not to be negative. There
 
      need be no sequencing of frame numbers either. Thus, a frame identification URL would look
 
      like distren://<servername>/job/<jobid>/frame/<frameid>
 
  - size: A frame hopefully represents a smaller unit of work in terms of the rendering
 
      back-end's capabilities. For example, POV-Ray's CLI can initiate and carry out the rendering
 
      of an entire animation. But distren would hopefully be able to provide a clean method for
 
      rendering each individual frame and bringing the resulting set of files back to the user
 
      in a much shorter time than a single computer could on its own. Many smaller and distinct
 
      is key to a project being benefited by distren.
 
  - dependencies: One cannot escape from frame's completion potentially requiring the completion
 
      of another. Thus, each frame's record must explicitly list the URLs of frames that need to
 
      be completed prior to the said frame. For a server to complete a frame dependent on other
 
      frames, those other frames must be transfered to the first server and made available.
 
  - packaging: To render a single frame and move it about somewhere is normally trivial. However,
 
      one frame or rendering unit of a given backend may produce multiple files. For this reason
 
      and for further uniformity and simplification, the data files representing one frame shall
 
      be transferred using the tarball format.
 

	
 
- client: A distren client is able to submit, query state of, and download completed frames of
 
      jobs registered in a server.
 
  - servers: A server, though having many more functions, shall be able to also perform the list
 
      of actions a client may perform.
 

	
 
- user: A user is an entity which is given access to a distren server.
 
  - user identifiction: As users are per-server, a user shall be identified by the name of the
 
      server he has an account on combined with his handle.
 
      distren://<servername>/user/<username> , like distren://ohnopub.net/user/ohnobinki
 

	
 
- file: There are different uses of files above and distributed rendering requires file distribution.
 
  - file identification: Every file mentioned above was at least in the context of a job. Thus,
 
      file identification numbers shall be assigned in the context of a job identification number.
 
      They shall, however, be numeric.
 
      distren://<servername>/job/<jobid>/file/<fileid>
doc/distrenjob.xml.example
Show inline comments
 
<?xml version="1.0" encoding="utf-8" ?>
 
<!--
 
  Copyright 2009 Nathan Phillip Brink, Ethan Zonca, Matthew Orlando
 

	
 
  This file is a part of DistRen.
 

	
 
  DistRen is free software: you can redistribute it and/or modify
 
  it under the terms of the GNU Affero General Public License as published by
 
  the Free Software Foundation, either version 3 of the License, or
 
  (at your option) any later version.
 

	
 
  DistRen is distributed in the hope that it will be useful,
 
  but WITHOUT ANY WARRANTY; without even the implied warranty of
 
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 
  GNU Affero General Public License for more details.
 

	
 
  You should have received a copy of the GNU Affero General Public License
 
  along with DistRen.  If not, see <http://www.gnu.org/licenses/>.
 
-->
 

	
 
<distren>
 
  <!-- is the priority option actually useful? -->
 
  <job
 
     name="distren://ohnopub.net/job/36"
 
     submitter="distren://ohnopub.net/user/binki"
 
     type="povray"
 
     >
 

	
 
    <!--
 
	This tag is information that does not have to be present in
 
	the original distrenjob.xml. For example, a povray's .ini file
 
	may already specify this resolution information. However,
 
	information like this should be extracted because this
 
	metadata may prove useful within distren's code. The <frames
 
	/> tag is a better example, as povray's .ini file will specify
 
	the information necessary for a sequence of frames to be
 
	rendered and we need to actually intercept/override that
 
	information.
 
    -->
 
    <resolution
 
       width="800"
 
       height="600"
 
       />
 

	
 
    <!--
 
	Currently, for simplicity, there will be a <frame /> tag for
 
	every frame to be rendered.
 
	every frame to be rendered. However, only the server from
 
	which the job originates will see every such frame.
 
	
 
	When a server requests a frame, it shall receive a full
 
	distrenjob.xml that contains only one frame. It will thus have
 
	enough information to render that one frame and return its
 
	results to the originating server.
 
    -->
 
    <frames>
 
      <frame name="distren://ohnopub.net/job/36/frame/23" />
 

	
 
      <!--
 
	  As described in architecture.txt, we allow frame
 
	  dependencies.
 
      -->
 
      <frame name="distren://ohnopub.net/job/36/frame/22">
 
	<dep name="distren://ohnopub.net/job/36/frame/23" />
 
	<dep name="distren://ohnopub.net/job/36/frame/24" />
 
      </frame>
 

	
 
      <frame name="distren://ohnopub.net/job/36/frame/24" />
 

	
 
      <!--
 
	  Assigned frames name their servers.
 
      -->
 
      <frame name="distren://ohnopub.net/job/36/frame/25"
 
	     server="distren://protofusion.org/" />
 

	
 
      <!--
 
	  Finished frames name their files and servers. Note that the
 
	  file's namespace is the job's URL. It is the job of the
 
	  server that renders a file to push it to its originating
 
	  server. It makes no sense for the originating server to try
 
	  to collect files manually ;-). Also, for simplicity, a frame
 
	  is considered assigned until the originating server obtains
 
	  a copy of it.
 

	
 
	  Notice how as a frame's status progresses, it only gains
 
	  information.
 
      -->
 
      <frame name="distren://ohnopub.net/job/36/frame/26"
 
	     server="distren://protofusion.org/"
 
	     file="distren://ohnopub.net/job/36/file/2" />
 
    </frames>
 

	
 
    <povray
 
	ini="sphere.ini"
 
	/>
 
  </job>
 
  
 
  <!--
 
      Additional section:
 
      The contents of this section should probably be
 
      moved to some central store, as it will be referenced
 
      in multiple jobs.
0 comments (0 inline, 0 general)