TimurH Posted November 3, 2008 Share Posted November 3, 2008 (edited) CGRU is a CG tools pack includes: Afanasy - free, open source render farm manager, RULES - web based CG projects tracker. Site: http://cgru.info Documentation https://cgru.readthedocs.io/en/latest/ Edited September 2, 2020 by TimurH Documentation link update. Quote Link to comment Share on other sites More sharing options...
TimurH Posted November 3, 2008 Author Share Posted November 3, 2008 (edited) ... Edited August 29, 2011 by TimurH Quote Link to comment Share on other sites More sharing options...
TimurH Posted November 3, 2008 Author Share Posted November 3, 2008 (edited) ... Edited August 29, 2011 by TimurH Quote Link to comment Share on other sites More sharing options...
TimurH Posted November 5, 2008 Author Share Posted November 5, 2008 Words spelling corrected (by a program). HTML mistakes fixed ( pages validated at "http://validator.w3.org"). Quote Link to comment Share on other sites More sharing options...
Jason Posted November 6, 2008 Share Posted November 6, 2008 Words spelling corrected (by a program). HTML mistakes fixed ( pages validated at "http://validator.w3.org"). Thanks for this great information! Quote Link to comment Share on other sites More sharing options...
paul Posted November 11, 2008 Share Posted November 11, 2008 hi - i've just had a look at this - and it looks v interesting! has anyone had a go yet? keen to hear of any experiances.... Quote Link to comment Share on other sites More sharing options...
Jason Posted November 11, 2008 Share Posted November 11, 2008 hi - i've just had a look at this - and it looks v interesting!has anyone had a go yet? keen to hear of any experiances.... I wonder what size site it's been tested on? In the 10's of procs? 100's? 1000's? Quote Link to comment Share on other sites More sharing options...
TimurH Posted November 11, 2008 Author Share Posted November 11, 2008 (edited) ... Edited November 4, 2010 by TimurH Quote Link to comment Share on other sites More sharing options...
TimurH Posted March 9, 2009 Author Share Posted March 9, 2009 (edited) ... Edited August 29, 2011 by TimurH Quote Link to comment Share on other sites More sharing options...
TimurH Posted September 14, 2009 Author Share Posted September 14, 2009 (edited) Last enhancements: Watch (Afanasy GUI) now uses "Qt Model/View Programming" - it allows to navigate jobs with 1 000 000 tasks. Multiply Tasks Renders: Render can run several tasks at the same time. Capacity mechanism: Each host and task has a capacity value. Task checks available (free) capacity on render, and if its enough, task launches and take its capacity value from render. Useful for farm with hosts and tasks with different "power". More powerful machine can run more tasks. More lite tasks can be executed on one machine. More balanced farm resources usage. No time wasting (time wasting may happen when client run some lite task which take no resources, but can't do to something else simultaneity, as server interpret it as "busy" or "free" only, server does not understand how much client busy). Tasks can variate capacity 'on-the-fly': Capacity variation can be described through its minimum and maximum coefficients (or may be simple constant). If task with 1000 capacity units runs on render with 4000 free capacity, task can take capacity coefficient 4. This coefficient can be caught in task command line (so Mantra can be launched 'mantra -j 4 ...', or 3000 capacity can be used by one 'houdini' task, for example to generate something, which use only one CPU, and 'mantra -j 3'). Hosts capacity values described in 'Farm Setup' ('xml' file). Edited December 8, 2010 by TimurH Quote Link to comment Share on other sites More sharing options...
TimurH Posted December 8, 2010 Author Share Posted December 8, 2010 (edited) Version 1.3.0 released. Last enhancements: Paths Map: Every client can describe how to translate its local paths to 'server' paths. On jobs send client translates all paths to 'server'. When client receives a task from server it translates all paths from 'sever' to local. With this feature you can setup Multi-OS render farm (and workstations). An artist can submit a job from any OS and it can be rendered on all OS-es. Python output filters(parsers) can produce error or warning. During output parsing you can find any error message and stop render process if it still running. Disk IO operations meter. Custom render host resource(s) meter(s). You can describe any resource meter by Python class and GUI will plot the graph. Lost of bug-fixes for MS Windows. Windows clients and GUI has the same functionality as Linux version. But there is still no server for MS Windows. Windows Must Die: You can list windows names in a file. Render will search windows with specified names and close them (sends WM_CLOSE signal). It is useful for MS Windows farm, as this OS when something going wrong often raise messages with "apologizes" which can hang render process until somebody press OK button. File "windowsmustdie.txt" example: mantra.exe mantra.exe - Application Error Movies creation tools. GUI and command line tools which can create 'Dailies' with special annotations and slate frame. http://cgru.sourceforge.net/utilities/moviemaker/doc/index.html You can render movies locally or send jobs to Afanasy farm. This tools are written on Python, GUI - PyQt, frame annotation - ImageMagick, encoding - ffmpeg. SoftImage XSI and 3D Studio MAX submission scripts. Every client sends version to server. GUI show clients versions. This version is a simple 'versions.txt' file contents and may be customized as you want. Edited August 29, 2011 by TimurH Quote Link to comment Share on other sites More sharing options...
TimurH Posted August 29, 2011 Author Share Posted August 29, 2011 Version 1.5.0 released. Last enhancements: System Job: Post commands are executed by a special system job. You can control it like any other common job. Post commands usually used to cleanup temporary files on job deletion. Errors Forgive Time: You can configure a time form last host error to exclude it from error (avoid) hosts list. Job Lifetime: Jobs can be deleted automatically after some time. Farm Services Limits: You can describe maximum tasks type (services) starts on all clients. Needed for software licenses. Python 3: You can construct and send jobs using Python 3. Maximum Tasks Per Host: Job will never start tasks on the same host above this limit. No matter how much free capacity it has. Sub-task Dependence: Tasks can depend on other task(s) sub-progress. They can run when some task finished partially. It is useful to render simulations. As simulations usually performed by a single task on a whole frame range. Now render tasks can run when the entire simulation is not finished. Houdini Submission Improvements: You can connect several Afanasy and other ROP nodes together to describe a complex job with dependences between ROPs. This job consists of a simulation with sub-task dependence. Two caches waiting the simulation, but cat run independently from each other. Mantra tile render which produces three blocks which wait all the cache. Two blocks for preview which can run independently but wait tile render tasks. One to convert EXR files to JPEGs and one to generate a preview movie form EXRs. Quote Link to comment Share on other sites More sharing options...
Jason Posted August 29, 2011 Share Posted August 29, 2011 Congratulations -- looks like a solid release! Good luck! 1 Quote Link to comment Share on other sites More sharing options...
TimurH Posted March 9, 2014 Author Share Posted March 9, 2014 (edited) CGRU v2 released Major cghanges in latest releases (since last post here): - CGRU GitHub project started. - New site cgru.info. - Afanasy server is available for MS Windows OS. - JSON protocol. Python API is the same, but now it just helps to constuct a valid JSON objects and send them. - Web GUI. Online demo. - All config files are based on JSON too. - Automatic Wake-On-Lan. - Automatic Nimby and Free depends on CPU percentage for some time. - CGRU Keeper to launch all it's tools and any other configured. - Python 3 full support (Keeper and Movie Maker PyQt/PiSide, Afanasy job API, task services and parsers on a render client side). - Task output parser can return activity parameter, GUIs will show and update it along with percentage (for example when stereo it can show left or right eye is processing). - Thumbnails: Task output parser can recognize output image(s) and generate thumbnails with a configurable command. - Afanasy server events: server can generate an event (for example job error or done) and push a task in a system job with all uts data. Client has a special service for it, it can constuct a command from event data (it also includes job user data) and launch this command. For you can configure afanasy to send emails on job done and error. - RULES: Another big project started in CGRU and it will be the main its part. Online demo. It is a project tracker. It has Web GUI and keeps all data on a file server along with projects/scenes/shots/sequences on a file server. There is no need to fill in RULES with projects/shots. Just describe where projects are located and what the structure is, rules how to find scenes, shots, sources and output. So it name comes out hee: describe rules - navigate file server. The project in its beginig but it already can:- Navigate projects, scenes, shots on a file server. This navigation will update thumbnails, show latest result sequences and movies. Helps artists to find sources.- Define tasks, assign artists on shots, place comments. Mark shots with any defined tags (key, cleanup, full3d, done, error, any custom). Specify percentage, see average percentage for scene/project.- Artists receive new (comments) for shots they assigned and any other they interested in and have subscribed.- Create Dailies: RULES always knows what is a project, scene, shot, company, logged user and so on. Simple clicking on sequence in output it sends a job in Afanasy. It annotate images (project, shot, version, artist, comments and so on), encode a movie and place it in a special location. Later RULES shows dailies (all or latest) for each shot.- Convert movies to other format, codec, fps, explode to images sequence. All can be done parallel via Afanasy. - View, annotate, comment jpeg sequences via a special web player.- Process any custom operations (commands) with files and folders via Afanasy. ( in Cinelab it also copying huge files and generates md5 ) Globally RULES is a web engine. It navgates a special folders stucture, collencts hierarchical data. Data can contain rules where to find a special assets, how to show them, change them. There is a 4 ready assets designed specially for CG VFX: Project, Scenes, Scene, Shot. But you can do anything on RULES web engine. Here is a forum asset for RULES, you can ask questions there. In our company we have a special RULES servers that just copy huge files, generate md5. Edited March 9, 2014 by TimurH Quote Link to comment Share on other sites More sharing options...
pbowmar Posted March 17, 2014 Share Posted March 17, 2014 Hi Timur, I'm just starting to play with this, looks fantastic! Sadly, I am hitting some (hopefully) minor errors. Is there a forum for questions on setting up Afanasy? I am happy to contribute my findings as well, there are a couple of minor mistakes in the docs (PYTHONPATH pointing to /opt/cgru/afanasy/plugins/houdini which should actually be /opt/cgru/plugins/houdini, for example) but I am now stuck on an AVOIDING HOSTS issue that I can't figure out. Cheers, Peter B Quote Link to comment Share on other sites More sharing options...
TimurH Posted April 28, 2014 Author Share Posted April 28, 2014 (edited) Hi. Sorry for a long delay, but i am visiting this forum just periodically. You can ask questions on github, where the source code is hosted: https://github.com/CGRU/cgru/issues?direction=desc&sort=updated&state=open Now i making forum based on RULES web engine (it is a project tracker in CGRU (for now young and simple project tracker)) http://cgru.info/rules/#/Ask_Questions_Here/AFANASY ( soon i'll try to attract all users there, and github will be only for working source code ) btw http://cgru.info/ site (where is Contacts page) was not work for about a week for about a half of the world, now it moved to another server and should work for everybody Edited April 28, 2014 by TimurH Quote Link to comment Share on other sites More sharing options...
TimurH Posted April 28, 2014 Author Share Posted April 28, 2014 (edited) As for AVOIDING HOSTS: If task finished with an error on some host, the host is pushed to an error hosts list. Each job block and task has such list. If the same host produced more that 3 errors, job (task) will avoid it. This value (3) can be customized for each job. By default it is "-1" for jobs, in this case this value taken from user. The default value for user is 3 and can be customized too. You can reset this list from GUI - "Reset Error Hosts" item in a job RMB menu. Edited April 28, 2014 by TimurH Quote Link to comment Share on other sites More sharing options...
pbowmar Posted May 1, 2014 Share Posted May 1, 2014 Thanks Timur, I have also used the RULES to ask questions I have gotten it all working with Houdini now, it's very cool. Cheers, Peter B Quote Link to comment Share on other sites More sharing options...
TimurH Posted January 29, 2017 Author Share Posted January 29, 2017 Hi. Now Afanasy got a common forum: http://forum.cgru.info/viewforum.php?f=19 Quote Link to comment Share on other sites More sharing options...
TimurH Posted September 13, 2020 Author Share Posted September 13, 2020 (edited) Version 3.0.0 Released. Pools and Tickets You can set root pool NUKE:20 pool tickets to limit Nuke licenses on the entire farm. You can set some pool MEM:64 host tickets to limit RAM. And set corresponding tickets to your job blocks. So you will limil 20 NUKE tasks on the entire farm. And each host can run 1 task with 64 MEM tickets, 2 tasks with 32, or 1*32 and 2*16 on each host. Documentation moved on ReadTheDocs Changes log: https://cgru.readthedocs.io/en/latest/changes_log/changes_log.html#v3-0-0 Step-by-step Houdini distribute simulation guide: https://cgru.readthedocs.io/en/latest/software/houdini.html#distributed-simulations Edited September 13, 2020 by TimurH 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.